ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-582

ZooKeeper can revert to old data when a snapshot is created outside of normal processing

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.1.1, 3.2.1
    • Fix Version/s: 3.1.2, 3.2.2, 3.3.0
    • Component/s: server
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      fixed bug in zookeeper that can lead zookeeper to revert to old data when a snapshot is created without a corresponding log.

      Description

      when zookeeper starts up it will restore the most recent state (latest zxid) it finds in the data directory. unfortunately, in the quorum version of zookeeper updates are logged using an epoch based on the latest log file in a directory. if there is a snapshot with a higher epoch than the log files, the zookeeper server will start logging using an epoch one higher than the highest log file.

      so if a data directory has a snapshot with an epoch of 27 and there are no log files, zookeeper will start logging changes using epoch 1. if the cluster restarts the state will be restored from the snapshot with the epoch of 27, which in effect, restores old data.

      normal operation of zookeeper will never result in this situation.

      this does not effect standalone zookeeper.

      a fix should make sure to use an epoch one higher than the current state, whether it comes from the snapshot or log, and should include a sanity check to make sure that a follower never connects to a leader that has a lower epoch than its own.

      1. test.patch
        13 kB
        Benjamin Reed
      2. ZOOKEEPER-582.patch
        3 kB
        Mahadev konar
      3. ZOOKEEPER-582_3.2.patch
        3 kB
        Mahadev konar
      4. ZOOKEEPER-582.patch
        14 kB
        Mahadev konar
      5. ZOOKEEPER-582.patch
        14 kB
        Mahadev konar
      6. ZOOKEEPER-582.patch
        14 kB
        Mahadev konar
      7. ZOOKEEPER-582.patch
        14 kB
        Mahadev konar
      8. ZOOKEEPER-582_3.2.patch
        14 kB
        Mahadev konar
      9. ZOOKEEPER-582_3.1.patch
        13 kB
        Mahadev konar

        Activity

        Benjamin Reed created issue -
        Patrick Hunt made changes -
        Field Original Value New Value
        Fix Version/s 3.1.2 [ 12314394 ]
        Affects Version/s 3.1.1 [ 12313649 ]
        Priority Major [ 3 ] Blocker [ 1 ]
        Benjamin Reed made changes -
        Attachment test.patch [ 12425316 ]
        Mahadev konar made changes -
        Attachment ZOOKEEPER-582.patch [ 12425320 ]
        Mahadev konar made changes -
        Attachment ZOOKEEPER-582_3.2.patch [ 12425410 ]
        Mahadev konar made changes -
        Attachment ZOOKEEPER-582_3.1.patch [ 12425411 ]
        Mahadev konar made changes -
        Attachment ZOOKEEPER-582.patch [ 12425503 ]
        Mahadev konar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Patrick Hunt made changes -
        Assignee Mahadev konar [ mahadev ]
        Patrick Hunt made changes -
        Fix Version/s 3.3.0 [ 12313976 ]
        Mahadev konar made changes -
        Attachment ZOOKEEPER-582.patch [ 12425536 ]
        Mahadev konar made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Mahadev konar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Mahadev konar made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Mahadev konar made changes -
        Attachment ZOOKEEPER-582.patch [ 12425538 ]
        Mahadev konar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Benjamin Reed made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Mahadev konar made changes -
        Attachment ZOOKEEPER-582.patch [ 12425653 ]
        Mahadev konar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Mahadev konar made changes -
        Attachment ZOOKEEPER-582_3.2.patch [ 12425670 ]
        Mahadev konar made changes -
        Attachment ZOOKEEPER-582_3.1.patch [ 12425671 ]
        Mahadev konar made changes -
        Attachment ZOOKEEPER-582_3.1.patch [ 12425671 ]
        Mahadev konar made changes -
        Attachment ZOOKEEPER-582_3.1.patch [ 12425411 ]
        Mahadev konar made changes -
        Attachment ZOOKEEPER-582_3.1.patch [ 12425672 ]
        Mahadev konar made changes -
        Attachment ZOOKEEPER-582_3.1.patch [ 12425672 ]
        Mahadev konar made changes -
        Attachment ZOOKEEPER-582_3.1.patch [ 12425673 ]
        Benjamin Reed made changes -
        Hadoop Flags [Reviewed]
        Mahadev konar made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Release Note fixed bug in zookeeper that can lead zookeeper to revert to old data when a snapshot is created without a corresponding log.
        Resolution Fixed [ 1 ]
        Patrick Hunt made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Mahadev konar
            Reporter:
            Benjamin Reed
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development