ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-1156

Log truncation truncating log too much - can cause data loss

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.3.3
    • Fix Version/s: 3.3.4, 3.4.0
    • Component/s: quorum, server
    • Labels:
      None

      Description

      The log truncation relies on position calculation for a particular zxid to figure out the new size of the log file. There is a bug in PositionInputStream implementation which skips counting the bytes in the log which have value 0. This can lead to underestimating the actual log size. The log records which should be there can get truncated, leading to data loss on the participant which is executing the trunc.

      Clients can see different values depending on whether they connect to the node on which trunc was executed.

      1. ZOOKEEPER-1156.patch
        2 kB
        Vishal Kathuria

        Activity

        Hide
        Mahadev konar added a comment -

        Vishal,
        I am little confused. Why is there a correctness issue here? I can see performance issue but dont see a correctness issue. Can you explain the scenario?

        Show
        Mahadev konar added a comment - Vishal, I am little confused. Why is there a correctness issue here? I can see performance issue but dont see a correctness issue. Can you explain the scenario?
        Hide
        Vishal Kathuria added a comment -

        Here is the scenario

        Lets say the current leader A is at zxid 80.
        A participant B with zxid 81 joins and gets a message from leader TRUNC,80

        B then calculates the length of log up till zxid 80. The actual length is, say 450, but because of the bug, the value it calculates is 420. B truncates the log to size 420.

        When loadDatabase is called again, the log is replayed till 79 because log record 80 isn't complete.

        The node B doesn't have the change that had zxid 80. The leader will not send change 80 to B either.

        In my manual repro, the change with zxid 80 was a create. I could see the created node when I connected to A but not when connected to B.

        Show
        Vishal Kathuria added a comment - Here is the scenario Lets say the current leader A is at zxid 80. A participant B with zxid 81 joins and gets a message from leader TRUNC,80 B then calculates the length of log up till zxid 80. The actual length is, say 450, but because of the bug, the value it calculates is 420. B truncates the log to size 420. When loadDatabase is called again, the log is replayed till 79 because log record 80 isn't complete. The node B doesn't have the change that had zxid 80. The leader will not send change 80 to B either. In my manual repro, the change with zxid 80 was a create. I could see the created node when I connected to A but not when connected to B.
        Hide
        Vishal Kathuria added a comment -

        This patch includes the fix. It makes the following changes to PositionInputStream

        1. In read, use rc > -1 instead of rc > 0 so the bytes with 0 value do not get skipped.
        2. Overrided some functions related to marking. Idea is that if the user of this stream uses these functions, we shouldn't silently return incorrect position.

        Show
        Vishal Kathuria added a comment - This patch includes the fix. It makes the following changes to PositionInputStream 1. In read, use rc > -1 instead of rc > 0 so the bytes with 0 value do not get skipped. 2. Overrided some functions related to marking. Idea is that if the user of this stream uses these functions, we shouldn't silently return incorrect position.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12490846/ZOOKEEPER-1156.patch
        against trunk revision 1157698.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/465//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/465//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/465//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12490846/ZOOKEEPER-1156.patch against trunk revision 1157698. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/465//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/465//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/465//console This message is automatically generated.
        Hide
        Vishal Kathuria added a comment -

        Forgot to mention the testing for this fix. I have a test that I am writing for ZOOKEEPER-1154 that tests for this scenario as well.

        Show
        Vishal Kathuria added a comment - Forgot to mention the testing for this fix. I have a test that I am writing for ZOOKEEPER-1154 that tests for this scenario as well.
        Hide
        Hudson added a comment -

        Integrated in ZooKeeper-trunk #1293 (See https://builds.apache.org/job/ZooKeeper-trunk/1293/)
        ZOOKEEPER-1154, ZOOKEEPER-1156:
        Data inconsistency when the node(s) with the highest zxid is not present at the time of leader election
        Log truncation truncating log too much - can cause data loss

        Vishal Kathuria via camille

        camille : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164758
        Files :

        • /zookeeper/trunk/CHANGES.txt
        • /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java
        • /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/LearnerHandler.java
        • /zookeeper/trunk/src/java/test/org/apache/zookeeper/server/quorum/QuorumPeerMainTest.java
        Show
        Hudson added a comment - Integrated in ZooKeeper-trunk #1293 (See https://builds.apache.org/job/ZooKeeper-trunk/1293/ ) ZOOKEEPER-1154 , ZOOKEEPER-1156 : Data inconsistency when the node(s) with the highest zxid is not present at the time of leader election Log truncation truncating log too much - can cause data loss Vishal Kathuria via camille camille : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164758 Files : /zookeeper/trunk/CHANGES.txt /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/LearnerHandler.java /zookeeper/trunk/src/java/test/org/apache/zookeeper/server/quorum/QuorumPeerMainTest.java

          People

          • Assignee:
            Vishal Kathuria
            Reporter:
            Vishal Kathuria
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 24h
              24h
              Remaining:
              Remaining Estimate - 24h
              24h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development