[MESOS-10014] `tryUntrackFrameworkUnderRole` check failed in `HierarchicalAllocatorProcess::removeFramework`.

• Type: Bug
• Status: Accepted
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.10.0
• Component/s: master, test
• Labels: flaky-test foundations
• Assignee: Unassigned
• Reporter: Andrei Budnik
• Created: 18/Oct/19 10:52
• Updated: 19/Mar/20 22:35

Description

`ContentType/OperationReconciliationTest.AgentPendingOperationAfterMasterFailover/0` test failed:

F1018 09:05:14.310616 21391 hierarchical.cpp:745] Check failed: tryUntrackFrameworkUnderRole(framework, role)  Framework: e6284079-cb6a-4a47-8f9a-ea9b84ff622a-0000 role: default-role
*** Check failure stack trace: ***
    @     0x7f40fff0a1f6  google::LogMessage::Fail()
    @     0x7f40fff0a14f  google::LogMessage::SendToLog()
    @     0x7f40fff09a91  google::LogMessage::Flush()
    @     0x7f40fff0d12f  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f410fd828ac  mesos::internal::master::allocator::internal::HierarchicalAllocatorProcess::removeFramework()
    @          0x186b29f  _ZZN7process8dispatchIN5mesos8internal6master9allocator21MesosAllocatorProcessERKNS1_11FrameworkIDES8_EEvRKNS_3PIDIT_EEMSA_FvT0_EOT1_ENKUlOS6_PNS_11ProcessBaseEE_clESJ_SL_
    @          0x189c273  _ZN5cpp176invokeIZN7process8dispatchIN5mesos8internal6master9allocator21MesosAllocatorProcessERKNS3_11FrameworkIDESA_EEvRKNS1_3PIDIT_EEMSC_FvT0_EOT1_EUlOS8_PNS1_11ProcessBaseEE_JS8_SN_EEEDTclcl7forwardISC_Efp_Espcl7forwardIT0_Efp0_EEEOSC_DpOSP_
    @          0x18990b7  _ZN6lambda8internal7PartialIZN7process8dispatchIN5mesos8internal6master9allocator21MesosAllocatorProcessERKNS4_11FrameworkIDESB_EEvRKNS2_3PIDIT_EEMSD_FvT0_EOT1_EUlOS9_PNS2_11ProcessBaseEE_JS9_St12_PlaceholderILi1EEEE13invoke_expandISP_St5tupleIJS9_SR_EESU_IJOSO_EEJLm0ELm1EEEEDTcl6invokecl7forwardISD_Efp_Espcl6expandcl3getIXT2_EEcl7forwardISH_Efp0_EEcl7forwardISK_Efp2_EEEEOSD_OSH_N5cpp1416integer_sequenceImJXspT2_EEEESL_
    @          0x1896100  _ZNO6lambda8internal7PartialIZN7process8dispatchIN5mesos8internal6master9allocator21MesosAllocatorProcessERKNS4_11FrameworkIDESB_EEvRKNS2_3PIDIT_EEMSD_FvT0_EOT1_EUlOS9_PNS2_11ProcessBaseEE_IS9_St12_PlaceholderILi1EEEEclIISO_EEEDTcl13invoke_expandcl4movedtdefpT1fEcl4movedtdefpT10bound_argsEcvN5cpp1416integer_sequenceImILm0ELm1EEEE_Ecl16forward_as_tuplespcl7forwardIT_Efp_EEEEDpOSX_
    @          0x1895174  _ZN5cpp176invokeIN6lambda8internal7PartialIZN7process8dispatchIN5mesos8internal6master9allocator21MesosAllocatorProcessERKNS6_11FrameworkIDESD_EEvRKNS4_3PIDIT_EEMSF_FvT0_EOT1_EUlOSB_PNS4_11ProcessBaseEE_ISB_St12_PlaceholderILi1EEEEEISQ_EEEDTclcl7forwardISF_Efp_Espcl7forwardIT0_Efp0_EEEOSF_DpOSV_
    @          0x1894b2b  _ZN6lambda8internal6InvokeIvEclINS0_7PartialIZN7process8dispatchIN5mesos8internal6master9allocator21MesosAllocatorProcessERKNS7_11FrameworkIDESE_EEvRKNS5_3PIDIT_EEMSG_FvT0_EOT1_EUlOSC_PNS5_11ProcessBaseEE_JSC_St12_PlaceholderILi1EEEEEJSR_EEEvOSG_DpOT0_
    @          0x18943bc  _ZNO6lambda12CallableOnceIFvPN7process11ProcessBaseEEE10CallableFnINS_8internal7PartialIZNS1_8dispatchIN5mesos8internal6master9allocator21MesosAllocatorProcessERKNSA_11FrameworkIDESH_EEvRKNS1_3PIDIT_EEMSJ_FvT0_EOT1_EUlOSF_S3_E_ISF_St12_PlaceholderILi1EEEEEEclEOS3_
    @     0x7f41016deb22  _ZNO6lambda12CallableOnceIFvPN7process11ProcessBaseEEEclES3_
    @     0x7f410169620c  process::ProcessBase::consume()
    @     0x7f41016c0696  _ZNO7process13DispatchEvent7consumeEPNS_13EventConsumerE
    @          0x1822baa  process::ProcessBase::serve()
    @     0x7f4101692af1  process::ProcessManager::resume()
    @     0x7f410168ed68  _ZZN7process14ProcessManager12init_threadsEvENKUlvE_clEv
    @     0x7f41016b81e2  _ZNSt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEE9_M_invokeIJEEEvSt12_Index_tupleIJXspT_EEE
    @     0x7f41016b7244  _ZNSt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEclEv
    @     0x7f41016b6088  _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
    @     0x7f40fca44590  execute_native_thread_routine
    @     0x7f40ffa77e25  start_thread
    @     0x7f40fa396bad  __clone
    @              (nil)  (unknown)