ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-313

Problem with successive leader failures when no client is connected

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 3.0.0, 3.0.1
    • Fix Version/s: 3.1.1
    • Component/s: server
    • Labels:
      None
    • Environment:

      all

      Description

      Steps to reproduce:

      Create a 3 node cluster . Run some transactions and then stop all clients. Make sure no other clients connect for the duration of the test.

      Let L1 be the current leader. Bring down L1. Let L2 be the leader chosen. Let the third node be N3. Note that this will increase the txn id for N3's snapshot without any transaction being logged. Now bring up L1 – same will happen for L1. Now bring down L2.

      Both N3 and L1 now have snapshots with a transaction id greater than the last logged transaction. Whoever is elected leader will try to restore its state from the filesystem and fail.

      One easy workaround is obviously to change the FileTxnSnapLog not to save a snapshot if zxid > last logged zxid. The correct solution is possibly to log a transaction for leader election as well.

        Activity

        Hide
        Benjamin Reed added a comment -

        excellent find! thanx for the test case too! you are correct the new leader transaction is not being logged but should be. (actually according to our design spec, it must be.)

        Show
        Benjamin Reed added a comment - excellent find! thanx for the test case too! you are correct the new leader transaction is not being logged but should be. (actually according to our design spec, it must be.)
        Hide
        Mahadev konar added a comment -

        sunanda,
        can you try out our latest release 3.1.0? We had a bug in 3.0 and 3.0.1 - ZOOKEEPER-251

        which has been resolved in 3.1.

        I tried the above scenario on 3.1 but cannot reproduce it. I can reproduce that in 3.0.0 and 3.0.1
        Also, as ben mentioned even in 3.1 we do not log the new leader transaction whcih we should (this is not really incorrect but would be just to follow our design spec).

        Show
        Mahadev konar added a comment - sunanda, can you try out our latest release 3.1.0? We had a bug in 3.0 and 3.0.1 - ZOOKEEPER-251 which has been resolved in 3.1. I tried the above scenario on 3.1 but cannot reproduce it. I can reproduce that in 3.0.0 and 3.0.1 Also, as ben mentioned even in 3.1 we do not log the new leader transaction whcih we should (this is not really incorrect but would be just to follow our design spec).
        Hide
        Mahadev konar added a comment -

        sunanda,

        i am makring this as a duplicate of ZOOKEEPER-251. I have openened ZOOKEEPER-335 for logging the new leader election txn.
        Please feel free to reopen if that is not the case.

        Show
        Mahadev konar added a comment - sunanda, i am makring this as a duplicate of ZOOKEEPER-251 . I have openened ZOOKEEPER-335 for logging the new leader election txn. Please feel free to reopen if that is not the case.

          People

          • Assignee:
            Mahadev konar
            Reporter:
            Sunanda Bera
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development