Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-1489

Data loss after truncate on transaction log

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.4.3, 3.3.5
    • Fix Version/s: 3.3.6, 3.4.4, 3.5.0
    • Component/s: server
    • Labels:
      None
    • Environment:

      Tested on Ubuntu 12.04 and CentOS 6, should be reproducible elsewhere

    • Hadoop Flags:
      Reviewed

      Description

      The truncate method on the transaction log in the class org.apache.zookeeper.server.persistence.FileTxnLog will reduce the file size to the required amount without either closing or re-positioning the logStream (which could also be dangerous since the truncate method is not synchronized against concurrent writes to the log).

      This causes the next append to that log to create a small "hole" in the file which java would interpret as binary zeroes when reading it. This then causes to the FileTxnIterator.next() implementation to detect the end of the log file too early.

      I'll attach a small maven project with one junit test which can be used to reproduce the issue. Due to the blackbox nature of the test it will run for roughly 50 seconds unfortunately.

      Steps to reproduce:

      • Start an ensemble of zookeeper servers with at least 3 participants
      • Create one entry and the remove one of the servers from the ensemble temporarily (e.g. zk-2)
      • Create another entry which is hence only reflected on zk-1 and zk-3
      • Take zk-1 out of the ensemble without shutting it down (that is important, I did that by interrupting the network connection to that node) and clean zk-3
      • Bring back zk-2 and zk-3 so that they form a quorum
      • Allow zk-1 to connect again
      • zk-1 will receive a TRUNC message from zk-2 since zk-1 is now a minority knowing about that second node creation event
      • Create a third node
      • Force zk-1 to become master somehow
      • That third node will be gone

        Attachments

        1. ZOOKEEPER-1489.patch
          31 kB
          Patrick Hunt
        2. ZOOKEEPER-1489.patch
          32 kB
          Patrick Hunt
        3. ZOOKEEPER-1489.patch
          32 kB
          Patrick Hunt
        4. ZOOKEEPER-1489.patch
          34 kB
          Patrick Hunt
        5. ZOOKEEPER-1489_br34.patch
          31 kB
          Patrick Hunt
        6. ZOOKEEPER-1489_br34.patch
          31 kB
          Patrick Hunt
        7. ZOOKEEPER-1489_br34.patch
          32 kB
          Patrick Hunt
        8. ZOOKEEPER-1489_br34.patch
          33 kB
          Patrick Hunt
        9. ZOOKEEPER-1489_br33.patch
          29 kB
          Patrick Hunt
        10. ZOOKEEPER-1489_br33.patch
          30 kB
          Patrick Hunt
        11. ZOOKEEPER-1489_br33.patch
          30 kB
          Patrick Hunt
        12. ZOOKEEPER-1489_br33.patch
          31 kB
          Patrick Hunt
        13. TruncateTxLogCorruption.tgz
          7 kB
          Christian Ziech
        14. TruncateTxLogCorruption.tgz
          7 kB
          Christian Ziech

          Activity

            People

            • Assignee:
              phunt Patrick Hunt
              Reporter:
              christianz Christian Ziech
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: