Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-8601

Master crashes during slave reregistration after failover.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 1.5.0
    • 1.5.1, 1.6.0
    • master
    • Mesosphere Sprint 75
    • 3

    Description

      The following happened after a master failover.
      During slave reregistration, new tasks were added and the new leading master notified all of its subscribers, and triggered the following check failure:

      F0222 15:53:44.440387  2805 master.cpp:11190] Check failed: 'framework' Must be non NULL
      *** Check failure stack trace: ***
      @     0x7f1357be521d  google::LogMessage::Fail()
      @     0x7f1357be704d  google::LogMessage::SendToLog()
      @     0x7f1357be4e0c  google::LogMessage::Flush()
      @     0x7f1357be7949  google::LogMessageFatal::~LogMessageFatal()
      @     0x7f1356c80e2d  google::CheckNotNull<>()
      @     0x7f1356ce2666  mesos::internal::master::Master::Subscribers::send()
      @     0x7f1356cece83  mesos::internal::master::Slave::addTask()
      @     0x7f1356cf3206  mesos::internal::master::Slave::Slave()
      @     0x7f1356cf5b90  mesos::internal::master::Master::__reregisterSlave()
      @     0x7f1356d02cf8  mesos::internal::master::Master::_reregisterSlave()
      @     0x7f1357b43761  process::ProcessBase::consume()
      @     0x7f1357b5248c  process::ProcessManager::resume()
      @     0x7f1357b579f6  _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
      @     0x7f1354e6c230  (unknown)
      @     0x7f135468ae25  start_thread
      @     0x7f13543b834d  __clone
      

      This was because the master tried to get the framework info when sending the notification: https://github.com/apache/mesos/blob/1.5.x/src/master/master.cpp#L11190

      But it added the framework after that:
      https://github.com/apache/mesos/blob/1.5.x/src/master/master.cpp#L6963

      Attachments

        Issue Links

          Activity

            People

              chhsia0 Chun-Hung Hsiao
              chhsia0 Chun-Hung Hsiao
              Greg Mann Greg Mann
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: