[MESOS-9904] ExecutorHttpApiTest.HeartbeatCalls is flaky

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: flaky flaky-test
• Assignee: Unassigned
• Reporter: Qian Zhang
• Created: 24/Jul/19 02:41
• Updated: 24/Jul/19 08:52

Description

We observed a segfault in `ExecutorHttpApiTest.HeartbeatCalls`.

10:28:17 [ RUN      ] ExecutorHttpApiTest.HeartbeatCalls
10:28:17 I0522 10:28:17.185608  9748 executor.cpp:206] Version: 1.9.0
10:28:17 W0522 10:28:17.185626  9748 process.cpp:2829] Attempted to spawn already running process version@172.16.10.73:34162
10:28:17 I0522 10:28:17.186307  9751 executor.cpp:432] Connected with the agent
10:28:17 F0522 10:28:17.186410  9751 owned.hpp:112] Check failed: 'get()' Must be non NULL 
10:28:17 *** Check failure stack trace: ***
10:28:17     @     0x7f4107fe103d  google::LogMessage::Fail()
10:28:17     @     0x7f4107fe317d  google::LogMessage::SendToLog()
10:28:17     @     0x7f4107fe0ba8  google::LogMessage::Flush()
10:28:17     @     0x7f4107fe3be9  google::LogMessageFatal::~LogMessageFatal()
10:28:17     @     0x564bed480290  _ZNSt17_Function_handlerIFvvEZN5mesos8internal5tests39ExecutorHttpApiTest_HeartbeatCalls_Test8TestBodyEvEUlvE_E9_M_invokeERKSt9_Any_data
10:28:17     @     0x7f4106a24f70  process::AsyncExecutorProcess::execute<>()
10:28:17     @     0x7f4106a3277b  _ZNO6lambda12CallableOnceIFvPN7process11ProcessBaseEEE10CallableFnINS_8internal7PartialIZNS1_8dispatchI7NothingNS1_20AsyncExecutorProcessERKSt8functionIFvvEESG_EENS1_6FutureIT_EERKNS1_3PIDIT0_EEMSL_FSI_T1_EOT2_EUlSt10unique_ptrINS1_7PromiseISA_EESt14default_deleteISW_EEOSE_S3_E_JSZ_SE_St12_PlaceholderILi1EEEEEEclEOS3_
10:28:17     @     0x7f4107f240f1  process::ProcessBase::consume()
10:28:17     @     0x7f4107f39797  process::ProcessManager::resume()
10:28:17     @     0x7f4107f3ea06  _ZNSt6thread11_State_implISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
10:28:17     @     0x7f41081ef6ff  execute_native_thread_routine
10:28:17     @     0x7f410420baa1  start_thread
10:28:17     @     0x7f41035b0c4d  clone
10:28:17 The test binary has crashed OR the timeout has been exceeded!
10:28:17 ~/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-centos-6

Issue Links

• is related to: MESOS-8096 Enqueueing events in MockHTTPScheduler can lead to segfaults.
• relates to: MESOS-9560 ContentType/AgentAPITest.MarkResourceProviderGone/1 is flaky
• relates to: MESOS-9905 Two unit tests in `DefaultExecutorTest` are flaky