[MESOS-9579] ExecutorHttpApiTest.HeartbeatCalls is flaky.

• Type: Bug
• Status: Accepted
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.8.0
• Component/s: executor
• Labels: flaky flaky-test foundations

Environment

Centos 6

• Story Points: 2
• Assignee: Unassigned
• Reporter: Till Toenshoff
• Created: 15/Feb/19 23:14
• Updated: 20/Mar/19 17:18

Description

I just saw this failing on our internal CI:

21:42:35 [ RUN ] ExecutorHttpApiTest.HeartbeatCalls
21:42:35 I0215 21:42:35.917752 17173 executor.cpp:206] Version: 1.8.0
21:42:35 W0215 21:42:35.917771 17173 process.cpp:2829] Attempted to spawn already running process version@172.16.10.166:35439
21:42:35 I0215 21:42:35.918581 17174 executor.cpp:432] Connected with the agent
21:42:35 F0215 21:42:35.918857 17174 owned.hpp:112] Check failed: 'get()' Must be non NULL 
21:42:35 *** Check failure stack trace: ***
21:42:35 @ 0x7fb93ce1d1dd google::LogMessage::Fail()
21:42:35 @ 0x7fb93ce1ee7d google::LogMessage::SendToLog()
21:42:35 @ 0x7fb93ce1cdb3 google::LogMessage::Flush()
21:42:35 @ 0x7fb93ce1f879 google::LogMessageFatal::~LogMessageFatal()
21:42:35 @ 0x55e80a099f76 google::CheckNotNull<>()
21:42:35 @ 0x55e80a07dde4 _ZNSt17_Function_handlerIFvvEZN5mesos8internal5tests39ExecutorHttpApiTest_HeartbeatCalls_Test8TestBodyEvEUlvE_E9_M_invokeERKSt9_Any_data
21:42:35 @ 0x7fb93baea260 process::AsyncExecutorProcess::execute<>()
21:42:35 @ 0x7fb93baf62cb _ZNO6lambda12CallableOnceIFvPN7process11ProcessBaseEEE10CallableFnINS_8internal7PartialIZNS1_8dispatchI7NothingNS1_20AsyncExecutorProcessERKSt8functionIFvvEESG_EENS1_6FutureIT_EERKNS1_3PIDIT0_EEMSL_FSI_T1_EOT2_EUlSt10unique_ptrINS1_7PromiseISA_EESt14default_deleteISW_EEOSE_S3_E_JSZ_SE_St12_PlaceholderILi1EEEEEEclEOS3_
21:42:36 @ 0x7fb93cd646b1 process::ProcessBase::consume()
21:42:36 @ 0x7fb93cd794ba process::ProcessManager::resume()
21:42:36 @ 0x7fb93cd7d486 _ZNSt6thread11_State_implISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
21:42:36 @ 0x7fb93d02a1af execute_native_thread_routine
21:42:36 @ 0x7fb939794aa1 start_thread
21:42:36 @ 0x7fb938b39c4d clone
21:42:36 The test binary has crashed OR the timeout has been exceeded!
21:42:36 ~/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-centos-6
21:42:36 mkswap: /tmp/swapfile: warning: don't erase bootbits sectors
21:42:36 on whole disk. Use -f to force.
21:42:36 Setting up swapspace version 1, size = 8388604 KiB
21:42:36 no label, UUID=dda5aa26-dba6-4ac8-bc6c-41264f510694
21:42:36 gcc (GCC) 6.3.1 20170216 (Red Hat 6.3.1-3)
21:42:36 Copyright (C) 2016 Free Software Foundation, Inc.
21:42:36 This is free software; see the source for copying conditions. There is NO
21:42:36 warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
21:42:36 Docker version 1.7.1, build 786b29d
21:42:36 curl 7.61.1 (x86_64-redhat-linux-gnu) libcurl/7.61.1 OpenSSL/1.0.1e zlib/1.2.3 c-ares/1.14.0 libssh2/1.8.0 nghttp2/1.6.0
21:42:36 Release-Date: 2018-09-05