[MESOS-3397] sorter.cpp: Check failed: total.resources.contains(slaveId)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.24.0
• Fix Version/s: 0.24.2, 0.25.1, 0.26.1, 0.27.2, 0.28.0
• Assignee: Neil Conway
• Reporter: Yan Xu
• Created: 09/Sep/15 00:13
• Updated: 29/Feb/16 18:20
• Resolved: 29/Feb/16 18:20

Description

Observed in production.

F0908 23:21:10.635751  6884 sorter.cpp:213] Check failed: total.resources.contains(slaveId)
*** Check failure stack trace: ***
    @     0x7f772cdb10bd  google::LogMessage::Fail()
    @     0x7f772cdb2f04  google::LogMessage::SendToLog()
    @     0x7f772cdb0cac  google::LogMessage::Flush()
    @     0x7f772cdb37f9  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f772c8162d0  mesos::internal::master::allocator::DRFSorter::remove()
    @     0x7f772c6f61bc  mesos::internal::master::allocator::HierarchicalAllocatorProcess<>::removeFramework()
    @     0x7f772cd61f09  process::ProcessManager::resume()
    @     0x7f772cd6220f  process::internal::schedule()
    @     0x7f772ce73610  execute_native_thread_routine
    @     0x7f772bcb883d  start_thread
    @     0x7f772b4aafdd  clone

This is following a framework removal:

I0908 23:21:10.619640  6884 master.cpp:4261] Framework failover timeout, removing framework 20150813-182946-1685138442-5050-58479-0425 (Some Scheduler) at scheduler-3c50e28c-a0f4-4619-8ea0-b786744e6e54@x.y.z.a:33952

Issue Links

• is related to: MESOS-3719 Core dump on /teardown
• is related to: MESOS-4071 Master crash during framework teardown (Check failed: total.resources.contains(slaveId))