[MESOS-4730] test-framework exits with SIGABRT

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Labels: mesosphere test-framework

Environment

OSX 10.11.3 (El Cap).

• Assignee: Unassigned
• Reporter: Neil Conway
• Created: 22/Feb/16 01:01
• Updated: 26/Nov/18 12:45
• Resolved: 26/Nov/18 12:45

Description

Steps to repro:

1. Build mesos from git
2. ./src/mesos-master.sh --registry=in_memory
3. ./src/mesos-slave.sh --master=X:5050
4. ./src/test-framework --master=X:5050

Observed behavior:

$ ./src/test-framework --master=10.0.0.11:5050
I0221 16:55:39.760979 1933606912 sched.cpp:222] Version: 0.28.0
I0221 16:55:39.768154 2674688 sched.cpp:326] New master detected at master@10.0.0.11:5050
I0221 16:55:39.768378 2674688 sched.cpp:336] No credentials provided. Attempting to register without authentication
I0221 16:55:39.769650 2674688 sched.cpp:703] Framework registered with 227af8fe-56b7-4853-bc65-4076bd7be95d-0000
Registered!
Received offer 227af8fe-56b7-4853-bc65-4076bd7be95d-O0 with cpus(*):8; mem(*):15360; disk(*):233112; ports(*):[31000-32000]
Launching task 0 using offer 227af8fe-56b7-4853-bc65-4076bd7be95d-O0
Launching task 1 using offer 227af8fe-56b7-4853-bc65-4076bd7be95d-O0
Launching task 2 using offer 227af8fe-56b7-4853-bc65-4076bd7be95d-O0
Launching task 3 using offer 227af8fe-56b7-4853-bc65-4076bd7be95d-O0
Launching task 4 using offer 227af8fe-56b7-4853-bc65-4076bd7be95d-O0
Received offer 227af8fe-56b7-4853-bc65-4076bd7be95d-O1 with cpus(*):3; mem(*):14720; disk(*):233112; ports(*):[31000-32000]
Task 0 is in state TASK_FAILED
Aborting because task 0 is in unexpected state TASK_FAILED with reason 1 from source 1 with message 'Executor terminated'
I0221 16:55:49.532826 528384 sched.cpp:1937] Asked to abort the driver
I0221 16:55:49.532888 528384 sched.cpp:1173] Aborting framework '227af8fe-56b7-4853-bc65-4076bd7be95d-0000'
I0221 16:55:49.533144 1933606912 sched.cpp:1903] Asked to stop the driver

Content of agent's stderr file for the crashed executor:

ABORT: (../../../mesos/3rdparty/libprocess/src/subprocess.cpp:322): Failed to os::execvpe on path '/usr/local/libexec/mesos/mesos-containerizer': No such file or directory
*** Aborted at 1456102539 (unix time) try "date -d @1456102539" if you are using GNU date ***
PC: @     0x7fff9666f002 __pthread_kill
*** SIGABRT (@0x7fff9666f002) received by PID 2637 (TID 0x700000104000) stack trace: ***
    @     0x7fff926e6eaa _sigtramp
    @     0x7000000ff7e0 (unknown)
    @     0x7fff867d96e7 abort
    @        0x10e087010 _Abort()
    @        0x10e086e5b _Abort()
    @        0x1109a3938 process::childMain()
    @        0x1109b3269 _ZNSt3__128__invoke_void_return_wrapperIiE6__callIJRNS_6__bindIPFiRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEPPcSD_RK6OptionINS_8functionIFivEEEERKN7process10Subprocess2IO20InputFileDescriptorsERKNSN_21OutputFileDescriptorsEST_bPiEJSB_RSD_SX_SK_RSO_RSR_SZ_RKbRA2_iEEEEEEiDpOT_
    @        0x1109b267c _ZNSt3__110__function6__funcINS_6__bindIPFiRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEPPcSC_RK6OptionINS_8functionIFivEEEERKN7process10Subprocess2IO20InputFileDescriptorsERKNSM_21OutputFileDescriptorsESS_bPiEJSA_RSC_SW_SJ_RSN_RSQ_SY_RKbRA2_iEEENS6_IS13_EESF_EclEv
    @        0x10f815721 std::__1::function<>::operator()()
    @        0x1109a2f33 process::defaultClone()
    @        0x1109b0fdd _ZNSt3__128__invoke_void_return_wrapperIiE6__callIJRPFiRKNS_8functionIFivEEEES7_EEEiDpOT_
    @        0x1109b0e6c std::__1::__function::__func<>::operator()()
    @        0x1109ad3b7 std::__1::function<>::operator()()
    @        0x1109a0fd0 process::subprocess()
    @        0x10f814653 mesos::internal::slave::PosixLauncher::fork()
    @        0x10f658c7d mesos::internal::slave::MesosContainerizerProcess::__launch()::$_5::operator()()
    @        0x10f65d289 _ZZZNK7process9_DeferredIZN5mesos8internal5slave25MesosContainerizerProcess8__launchERKNS1_11ContainerIDERKNS1_12ExecutorInfoERKNSt3__112basic_stringIcNSB_11char_traitsIcEENSB_9allocatorIcEEEERK6OptionISH_ERKNS1_7SlaveIDERKNS_3PIDINS3_5SlaveEEEbRKNSB_4listISK_INS1_5slave19ContainerLaunchInfoEENSF_ISZ_EEEEE3$_5EcvNSB_8functionIFT_T0_EEEINS_6FutureIbEERKNSX_15ContainerLogger14SubprocessInfoEEEvENKUlS1H_E_clES1H_ENKUlvE_clEv
    @        0x10f65d254 _ZNSt3__128__invoke_void_return_wrapperIN7process6FutureIbEEE6__callIJRZZNKS1_9_DeferredIZN5mesos8internal5slave25MesosContainerizerProcess8__launchERKNS7_11ContainerIDERKNS7_12ExecutorInfoERKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEERK6OptionISM_ERKNS7_7SlaveIDERKNS1_3PIDINS9_5SlaveEEEbRKNS_4listISP_INS7_5slave19ContainerLaunchInfoEENSK_IS14_EEEEE3$_5EcvNS_8functionIFT_T0_EEEIS3_RKNS12_15ContainerLogger14SubprocessInfoEEEvENKUlS1K_E_clES1K_EUlvE_EEES3_DpOT_
    @        0x10f65cf83 _ZNSt3__110__function6__funcIZZNK7process9_DeferredIZN5mesos8internal5slave25MesosContainerizerProcess8__launchERKNS4_11ContainerIDERKNS4_12ExecutorInfoERKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEERK6OptionISJ_ERKNS4_7SlaveIDERKNS2_3PIDINS6_5SlaveEEEbRKNS_4listISM_INS4_5slave19ContainerLaunchInfoEENSH_IS11_EEEEE3$_5EcvNS_8functionIFT_T0_EEEINS2_6FutureIbEERKNSZ_15ContainerLogger14SubprocessInfoEEEvENKUlS1J_E_clES1J_EUlvE_NSH_IS1L_EEFS1F_vEEclEv
    @        0x10ec26510 std::__1::function<>::operator()()
    @        0x10ec26433 _ZZN7process8dispatchIbEENS_6FutureIT_EERKNS_4UPIDERKNSt3__18functionIFS3_vEEEENKUlPNS_11ProcessBaseEE_clESE_
    @        0x10ec263f0 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN7process8dispatchIbEENS3_6FutureIT_EERKNS3_4UPIDERKNS_8functionIFS7_vEEEEUlPNS3_11ProcessBaseEE_SH_EEEvDpOT_
    @        0x10ec2608c _ZNSt3__110__function6__funcIZN7process8dispatchIbEENS2_6FutureIT_EERKNS2_4UPIDERKNS_8functionIFS6_vEEEEUlPNS2_11ProcessBaseEE_NS_9allocatorISH_EEFvSG_EEclEOSG_
    @        0x1108c73a2 std::__1::function<>::operator()()
    @        0x1108a106f process::ProcessBase::visit()
    @        0x1108f411e process::DispatchEvent::visit()
    @        0x10e0cbf71 process::ProcessBase::serve()
    @        0x11089dd61 process::ProcessManager::resume()
    @        0x1108a961f process::ProcessManager::init_threads()::$_1::operator()()
    @        0x1108a92a2 _ZNSt3__114__thread_proxyINS_5tupleIJNS_6__bindIZN7process14ProcessManager12init_threadsEvE3$_1JNS_17reference_wrapperIKNS_6atomicIbEEEEEEEEEEEEPvSD_
    @     0x7fff90186c13 _pthread_body
    @     0x7fff90186b90 _pthread_start

Issue Links

• is related to: MESOS-4867 Consider suggesting `make install` in "Getting Started"