Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-3280

Master fails to access replicated log after network partition

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.23.0
    • 0.26.0
    • master, replicated log

    • Zookeeper version 3.4.5--1

    • 8

    Description

      In a 5 node cluster with 3 masters and 2 slaves, and ZK on each node, when a network partition is forced, all the masters apparently lose access to their replicated log. The leading master halts. Unknown reasons, but presumably related to replicated log access. The others fail to recover from the replicated log. Unknown reasons. This could have to do with ZK setup, but it might also be a Mesos bug.

      This was observed in a Chronos test drive scenario described in detail here:
      https://github.com/mesos/chronos/issues/511

      With setup instructions here:
      https://github.com/mesos/chronos/issues/508

      Attachments

        1. rep-log-startup-race-test-1.patch
          4 kB
          Neil Conway
        2. rep-log-race-cond-logs.tar.gz
          20 kB
          Neil Conway

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. MESOS-3532 3 Master HA setup restarts every 3 minutes

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. MESOS-1399 Add retries for co-ordinator election.

          • Major - Major loss of function.
          • Accepted

          Activity

            People

              neilc Neil Conway
              bernd-mesos Bernd Mathiske
              Joris Van Remoortere Joris Van Remoortere
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: