[MESOS-8614] DefaultExecutorTests occassionally crash in the V1 Scheduler code

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.5.0
• Labels: flaky-test mesosphere
• Assignee: Unassigned
• Reporter: Chun-Hung Hsiao
• Created: 26/Feb/18 22:02
• Updated: 23/Apr/18 20:35
• Resolved: 23/Apr/18 20:35

Description

Occasionally the DefaultExecutorTest.ResourceLimitation/0 and DefaultExecutorTest.ROOT_ContainerStatusForTask/0 would crash with the following logs:

I*** Aborted at 1519639358 (unix time) try "date -d @1519639358" if you are using GNU date ***
0226 10:02:38.030114 21366 task_status_update_manager.cpp:538] Cleaning up status update stream for task a332e0b5-a713-47b2-85d8-358ce6a4118a of framework 507afc07-d395-4e76-aa11-4562ae07a9b3-0000
I0226 10:02:38.029911 21370 gc.cpp:90] Scheduling '/tmp/ROOT_DOCKER_DockerAndMesosContainerizers_DefaultExecutorTest_ResourceLimitation_0_UVxsKT/slaves/507afc07-d395-4e76-aa11-4562ae07a9b3-S0/frameworks/507afc07-d395-4e76-aa11-4562ae07a9b3-0000/executors/default/runs/37678c9e-fc27-40fa-8d26-b540ff88a381' for gc 6.99999968157333days in the future
I0226 10:02:38.030480 21370 gc.cpp:90] Scheduling '/tmp/ROOT_DOCKER_DockerAndMesosContainerizers_DefaultExecutorTest_ResourceLimitation_0_UVxsKT/slaves/507afc07-d395-4e76-aa11-4562ae07a9b3-S0/frameworks/507afc07-d395-4e76-aa11-4562ae07a9b3-0000/executors/default' for gc 6.99999968157333days in the future
I0226 10:02:38.030591 21370 gc.cpp:90] Scheduling '/tmp/ROOT_DOCKER_DockerAndMesosContainerizers_DefaultExecutorTest_ResourceLimitation_0_UVxsKT/slaves/507afc07-d395-4e76-aa11-4562ae07a9b3-S0/frameworks/507afc07-d395-4e76-aa11-4562ae07a9b3-0000' for gc 6.99999968157333days in the future
PC: @     0x7f9b6df74eb3 mesos::v1::scheduler::Mesos::send()
*** SIGSEGV (@0x0) received by PID 32110 (TID 0x7f9b626a9700) from PID 0; stack trace: ***
    @     0x7f9b3717b9c2 (unknown)
    @     0x7f9b37180689 (unknown)
    @     0x7f9b371743e8 (unknown)
    @     0x7f9b6b7d3670 (unknown)
    @     0x7f9b6df74eb3 mesos::v1::scheduler::Mesos::send()
    @     0x55a24270c0f6 _ZNK5mesos8internal5tests2v19scheduler23SendAcknowledgeActionP2INS_2v111FrameworkIDENS5_7AgentIDEE10gmock_ImplIFvPNS5_9scheduler5MesosERKNSA_12Event_UpdateEEE17gmock_PerformImplISC_SF_N7testing8internal12ExcessiveArgESL_SL_SL_SL_SL_SL_SL_EEvRKSt5tupleIJSC_SF_EET_T0_T1_T2_T3_T4_T5_T6_T7_T8_
    @     0x55a24270c26a _ZN5mesos8internal5tests2v19scheduler23SendAcknowledgeActionP2INS_2v111FrameworkIDENS5_7AgentIDEE10gmock_ImplIFvPNS5_9scheduler5MesosERKNSA_12Event_UpdateEEE7PerformERKSt5tupleIJSC_SF_EE
    @     0x55a2425fcc1e _ZN7testing8internal12DoBothActionI17PromiseArgActionPILi1EPN7process7PromiseIN5mesos2v19scheduler12Event_UpdateEEEENS5_8internal5tests2v19scheduler23SendAcknowledgeActionP2INS6_11FrameworkIDENS6_7AgentIDEEEE4ImplIFvPNS7_5MesosERKS8_EE7PerformERKSt5tupleIJSN_SP_EE
    @     0x55a24262e2b7 testing::internal::FunctionMockerBase<>::UntypedPerformAction()
    @     0x55a2438a2d19 testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith()
    @     0x55a24270f27a mesos::internal::tests::scheduler::MockHTTPScheduler<>::events()
    @     0x55a24268aae3 std::_Function_handler<>::_M_invoke()
    @     0x7f9b6df78bf8 process::AsyncExecutorProcess::execute<>()
    @     0x7f9b6df8155d _ZNO6lambda12CallableOnceIFvPN7process11ProcessBaseEEE10CallableFnINS_8internal7PartialIZNS1_8dispatchI7NothingNS1_20AsyncExecutorProcessERKSt8functionIFvRKSt5queueIN5mesos2v19scheduler5EventESt5dequeISH_SaISH_EEEEESL_SR_RSL_EENS1_6FutureIT_EERKNS1_3PIDIT0_EEMSX_FSU_T1_T2_EOT3_OT4_EUlSt10unique_ptrINS1_7PromiseISA_EESt14default_deleteIS1B_EEOSP_OSL_S3_E_JS1E_SP_SL_St12_PlaceholderILi1EEEEEEclEOS3_
    @     0x7f9b6eb3c1f1 process::ProcessBase::consume()
    @     0x7f9b6eb4eea2 process::ProcessManager::resume()
    @     0x7f9b6eb52bb6 _ZNSt6thread11_State_implISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
    @     0x7f9b6bcb283f (unknown)
    @     0x7f9b6b7c96da start_thread
    @     0x7f9b6b503d7f (unknown)

Attached logs of 3 crash instances.

Issue Links

• duplicates: MESOS-8096 Enqueueing events in MockHTTPScheduler can lead to segfaults.
• is a clone of: MESOS-8540 ROOT_TasksSharingViaSandboxVolumes is flaky.
• is related to: MESOS-8096 Enqueueing events in MockHTTPScheduler can lead to segfaults.