[MESOS-8278] Mesos Containerizer cannot recover due to check failure.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: containerization
• Labels: containerizer csi-post-mvp standalone
• Epic Link: Standalone Container
• Assignee: Unassigned
• Reporter: Gilbert Song
• Created: 29/Nov/17 23:14
• Updated: 20/Jan/18 01:35
• Resolved: 20/Jan/18 01:35

Description

Mesos containerizer cannot recover due to a check failure on nested container's sandbox directory.

I1129 22:00:42.556479  5812 containerizer.cpp:670] Recovering containerizer
F1129 22:00:42.560739  5812 containerizer.cpp:912] CHECK_SOME(directory): is NONE 
*** Check failure stack trace: ***
    @     0x7f7e6cf1294d  google::LogMessage::Fail()
    @     0x7f7e6cf11d1e  google::LogMessage::SendToLog()
    @     0x7f7e6cf1261d  google::LogMessage::Flush()
    @     0x7f7e6cf15a98  google::LogMessageFatal::~LogMessageFatal()
    @     0x55ca72a95197  _CheckFatal::~_CheckFatal()
    @     0x7f7e6bb23770  mesos::internal::slave::MesosContainerizerProcess::recover()
    @     0x7f7e6bbe643c  _ZZN7process8dispatchI7NothingN5mesos8internal5slave25MesosContainerizerProcessERK6OptionINS4_5state10SlaveStateEESB_EENS_6FutureIT_EERKNS_3PIDIT0_EEMSG_FSE_T1_EOT2_ENKUlRS9_PNS_11ProcessBaseEE_clESP_SR_
    @     0x7f7e6bbe6295  _ZNSt5_BindIFZN7process8dispatchI7NothingN5mesos8internal5slave25MesosContainerizerProcessERK6OptionINS5_5state10SlaveStateEESC_EENS0_6FutureIT_EERKNS0_3PIDIT0_EEMSH_FSF_T1_EOT2_EUlRSA_PNS0_11ProcessBaseEE_SA_St12_PlaceholderILi1EEEE6__callIvJOSS_EJLm0ELm1EEEESE_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
    @     0x7f7e6bbe61f6  _ZNSt5_BindIFZN7process8dispatchI7NothingN5mesos8internal5slave25MesosContainerizerProcessERK6OptionINS5_5state10SlaveStateEESC_EENS0_6FutureIT_EERKNS0_3PIDIT0_EEMSH_FSF_T1_EOT2_EUlRSA_PNS0_11ProcessBaseEE_SA_St12_PlaceholderILi1EEEEclIJSS_EvEESH_DpOT_
    @     0x7f7e6bbe5f02  _ZNSt17_Function_handlerIFvPN7process11ProcessBaseEESt5_BindIFZNS0_8dispatchI7NothingN5mesos8internal5slave25MesosContainerizerProcessERK6OptionINS9_5state10SlaveStateEESG_EENS0_6FutureIT_EERKNS0_3PIDIT0_EEMSL_FSJ_T1_EOT2_EUlRSE_S2_E_SE_St12_PlaceholderILi1EEEEE9_M_invokeERKSt9_Any_dataOS2_
    @     0x7f7e6ce37cf4  std::function<>::operator()()
    @     0x7f7e6ce1ded4  process::ProcessBase::visit()
    @     0x7f7e6cea38fe  process::DispatchEvent::visit()
    @     0x7f7e6a9741b1  process::ProcessBase::serve()
    @     0x7f7e6ce1a8eb  process::ProcessManager::resume()
    @     0x7f7e6ce2b86e  process::ProcessManager::init_threads()::$_7::operator()()
    @     0x7f7e6ce2b715  _ZNSt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvE3$_7vEE9_M_invokeIJEEEvSt12_Index_tupleIJXspT_EEE
    @     0x7f7e6ce2b6e5  std::_Bind_simple<>::operator()()
    @     0x7f7e6ce2b6bc  std::thread::_Impl<>::_M_run()
    @     0x7f7e6617d030  (unknown)
    @     0x7f7e65c966aa  start_thread
    @     0x7f7e659cbe9d  (unknown)

Maybe related to the change of standalone container support.

Issue Links

• duplicates: MESOS-8416 CHECK failure if trying to recover nested containers but the framework checkpointing is not enabled.