[MESOS-3728] Libprocess: Flaky behavior on test suite when finalizing.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.26.0

Environment

OS 10.11.1 Beta (15B30a),
Apple LLVM version 7.0.0 (clang-700.0.72)

• Assignee: Benjamin Mahler
• Reporter: Till Toenshoff
• Shepherd: Jie Yu
• Created: 14/Oct/15 09:52
• Updated: 08/Nov/15 12:51
• Resolved: 16/Oct/15 00:10

Description

The issue manifests in the following stacktrace. Triggering the issue is not too hard on my machine - fails in more than 10% of all attempts.

[----------] Global test environment tear-down
[==========] 148 tests from 22 test cases ran. (1323 ms total)
[  PASSED  ] 148 tests.

  YOU HAVE 2 DISABLED TESTS

Assertion failed: (ec == 0), function unlock, file /BuildRoot/Library/Caches/com.apple.xbs/Sources/libcxx/libcxx-120.1/src/mutex.cpp, line 45.
*** Aborted at 1444816067 (unix time) try "date -d @1444816067" if you are using GNU date ***
PC: @     0x7fff94e370ae __pthread_kill
*** SIGABRT (@0x7fff94e370ae) received by PID 11537 (TID 0x700000104000) stack trace: ***
    @     0x7fff9410d52a _sigtramp
    @     0x7000001034d8 (unknown)
    @     0x7fff9c71237b abort
    @     0x7fff9c6d99c4 __assert_rtn
    @     0x7fff8a27afc8 std::__1::mutex::unlock()
    @        0x102b109a9 _ZZ11synchronizeINSt3__15mutexEE12SynchronizedIT_EPS3_ENKUlPS1_E0_clES6_
    @        0x102b10988 _ZZ11synchronizeINSt3__15mutexEE12SynchronizedIT_EPS3_ENUlPS1_E0_8__invokeES6_
    @        0x102b10a40 Synchronized<>::~Synchronized()
    @        0x102ad70f5 Synchronized<>::~Synchronized()
    @        0x102aeb45c Gate::empty()
    @        0x102ace2cc process::ProcessManager::wait()
    @        0x102ad22b6 process::wait()
    @        0x1024b47ee process::wait()
    @        0x1029b56a6 process::http::Connection::Data::~Data()
    @        0x1029b55d5 process::http::Connection::Data::~Data()
    @        0x1029a71bc std::__1::__shared_ptr_emplace<>::__on_zero_shared()
    @     0x7fff8a27acb8 std::__1::__shared_weak_count::__release_shared()
    @        0x1024ba50f std::__1::shared_ptr<>::~shared_ptr()
    @        0x1024ba4d5 std::__1::shared_ptr<>::~shared_ptr()
    @        0x1024ba4b5 process::http::Connection::~Connection()
    @        0x1024a2cc5 process::http::Connection::~Connection()
    @        0x10297b81d _ZZZN7process4http8internal7requestERKNS0_7RequestEbENK3$_2clENS0_10ConnectionEENKUlvE_clEv
    @        0x10297b6fd _ZN7process20AsyncExecutorProcess7executeIZZNS_4http8internal7requestERKNS2_7RequestEbENK3$_2clENS2_10ConnectionEEUlvE_EE7NothingRKT_PN5boost9enable_ifINSE_7is_voidINSt3__19result_ofIFSB_vEE4typeEEEvE4typeE
    @        0x10297d8be _ZZN7process8dispatchI7NothingNS_20AsyncExecutorProcessERKZZNS_4http8internal7requestERKNS3_7RequestEbENK3$_2clENS3_10ConnectionEEUlvE_PvSA_SD_EENS_6FutureIT_EERKNS_3PIDIT0_EEMSI_FSF_T1_T2_ET3_T4_ENKUlPNS_11ProcessBaseEE_clEST_
    @        0x10297d730 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN7process8dispatchI7NothingNS3_20AsyncExecutorProcessERKZZNS3_4http8internal7requestERKNS7_7RequestEbENK3$_2clENS7_10ConnectionEEUlvE_PvSE_SH_EENS3_6FutureIT_EERKNS3_3PIDIT0_EEMSM_FSJ_T1_T2_ET3_T4_EUlPNS3_11ProcessBaseEE_SX_EEEvDpOT_
    @        0x10297d3fc _ZNSt3__110__function6__funcIZN7process8dispatchI7NothingNS2_20AsyncExecutorProcessERKZZNS2_4http8internal7requestERKNS6_7RequestEbENK3$_2clENS6_10ConnectionEEUlvE_PvSD_SG_EENS2_6FutureIT_EERKNS2_3PIDIT0_EEMSL_FSI_T1_T2_ET3_T4_EUlPNS2_11ProcessBaseEE_NS_9allocatorISX_EEFvSW_EEclEOSW_
    @        0x102aec69f std::__1::function<>::operator()()
    @        0x102acef4f process::ProcessBase::visit()
    @        0x102b0f7de process::DispatchEvent::visit()
    @        0x1023417d1 process::ProcessBase::serve()
    @        0x102acbce1 process::ProcessManager::resume()
    @        0x102ad6a4c process::ProcessManager::init_threads()::$_1::operator()()
make[5]: *** [check-local] Abort trap: 6
make[4]: *** [check-am] Error 2
make[3]: *** [check-recursive] Error 1
make[2]: *** [check-recursive] Error 1
make[1]: *** [check] Error 2
make: *** [check-recursive] Error 1