ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-1448

Node+Quota creation in transaction log can crash leader startup

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 3.3.5
    • Fix Version/s: 3.4.6, 3.5.0
    • Component/s: server
    • Labels:
      None

      Description

      Hi,

      I've found a bug in zookeeper related to quota creation which can shutdown zookeeper leader on startup.

      Steps to reproduce:
      1. create /quota_bug
      2. setquota -n 10000 /quota_bug
      3. stop the whole ensemble (the previous operations should be in the transaction log)
      4. start all the servers
      5. the elected leader will shutdown with an exception (Missing stat node for count /zookeeper/quota/quota_bug/zookeeper_
      stats)

      I've debugged a bit what happening and I found the following problem:
      On startup each server loads the last snapshot and replays the last transaction log. While doing this it fills up the pTrie variable of the DataTree with the path of the nodes which have quota.
      After the leader is elected the leader servers loads the snapshot and last transaction log but it doesn't clean up the pTrie variable. This means it still contains the "/quota_bug" path. Now when the "create /quota_bug" is processed from the transaction log the DataTree already thinks that the quota nodes ("/zookeeper/quota/quota_bug/zookeeper_limits" and "/zookeeper/quota/quota_bug/zookeeper_stats") are created but those node creation actually comes later in the transaction log. This leads to the missing stat node exception.

      I think clearing the pTrie should solve this problem.

      1. ZOOKEEPER-1448-trunk.patch
        3 kB
        Flavio Junqueira
      2. ZOOKEEPER-1448.patch
        1 kB
        Botond Hejj
      3. ZOOKEEPER-1448.patch
        4 kB
        Botond Hejj
      4. ZOOKEEPER-1448.patch
        4 kB
        Botond Hejj
      5. ZOOKEEPER-1448.patch
        5 kB
        Botond Hejj
      6. ZOOKEEPER-1448.patch
        3 kB
        Botond Hejj
      7. ZOOKEEPER-1448_branch3.3.patch
        5 kB
        Botond Hejj

        Activity

        Hide
        Flavio Junqueira added a comment -

        Closing issues after releasing 3.4.6.

        Show
        Flavio Junqueira added a comment - Closing issues after releasing 3.4.6.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in ZooKeeper-trunk #2046 (See https://builds.apache.org/job/ZooKeeper-trunk/2046/)
        ZOOKEEPER-1448. Node+Quota creation in transaction log can crash leader startup (Botond Hejj via fpj) (fpj: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1520436)

        • /zookeeper/trunk/CHANGES.txt
        • /zookeeper/trunk/src/java/main/org/apache/zookeeper/common/PathTrie.java
        • /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/DataTree.java
        • /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/DataTreeTest.java
        Show
        Hudson added a comment - SUCCESS: Integrated in ZooKeeper-trunk #2046 (See https://builds.apache.org/job/ZooKeeper-trunk/2046/ ) ZOOKEEPER-1448 . Node+Quota creation in transaction log can crash leader startup (Botond Hejj via fpj) (fpj: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1520436 ) /zookeeper/trunk/CHANGES.txt /zookeeper/trunk/src/java/main/org/apache/zookeeper/common/PathTrie.java /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/DataTree.java /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/DataTreeTest.java
        Hide
        Flavio Junqueira added a comment -

        Trunk: Committed revision 1520436.

        Show
        Flavio Junqueira added a comment - Trunk: Committed revision 1520436.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12601682/ZOOKEEPER-1448-trunk.patch
        against trunk revision 1519655.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1556//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1556//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1556//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12601682/ZOOKEEPER-1448-trunk.patch against trunk revision 1519655. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1556//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1556//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1556//console This message is automatically generated.
        Hide
        Flavio Junqueira added a comment -

        b3.4: Committed revision 1520418.

        Show
        Flavio Junqueira added a comment - b3.4: Committed revision 1520418.
        Hide
        Flavio Junqueira added a comment - - edited

        The latest patch didn't apply cleanly, so I generated a new one.

        Show
        Flavio Junqueira added a comment - - edited The latest patch didn't apply cleanly, so I generated a new one.
        Hide
        Flavio Junqueira added a comment -

        +1, looks good. I'll check this in to trunk as well.

        Show
        Flavio Junqueira added a comment - +1, looks good. I'll check this in to trunk as well.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12600750/ZOOKEEPER-1448.patch
        against trunk revision 1516126.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1548//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1548//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1548//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12600750/ZOOKEEPER-1448.patch against trunk revision 1516126. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1548//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1548//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1548//console This message is automatically generated.
        Hide
        Flavio Junqueira added a comment -

        Thanks, Botond. I'm submitting the patch to let jenkins run it.

        Show
        Flavio Junqueira added a comment - Thanks, Botond. I'm submitting the patch to let jenkins run it.
        Hide
        Botond Hejj added a comment -

        I don't see any change in 3.5.0 for the DataTree class so I think it is applicable for 3.5.0 as well. Although I haven't tested it

        Show
        Botond Hejj added a comment - I don't see any change in 3.5.0 for the DataTree class so I think it is applicable for 3.5.0 as well. Although I haven't tested it
        Hide
        Botond Hejj added a comment -

        Updated unittest to remove log4j dependency

        Show
        Botond Hejj added a comment - Updated unittest to remove log4j dependency
        Hide
        Flavio Junqueira added a comment -

        If I can get a patch that applies to the 3.4 branch cleanly and removes the references to log4j, then I can include it in 3.4.6. I suppose this is also an issue for 3.5.0, no?

        Show
        Flavio Junqueira added a comment - If I can get a patch that applies to the 3.4 branch cleanly and removes the references to log4j, then I can include it in 3.4.6. I suppose this is also an issue for 3.5.0, no?
        Hide
        Mahadev konar added a comment -

        I looked at the patch. THe patch looks good except for the point that Pat mentioned above that its moving the test toward log4j than using slf4j. For now I am moving this out to 3.4.5 for getting it done right. Botond, if you have sometime would you please update the patch using slf4j in the testcase.

        Show
        Mahadev konar added a comment - I looked at the patch. THe patch looks good except for the point that Pat mentioned above that its moving the test toward log4j than using slf4j. For now I am moving this out to 3.4.5 for getting it done right. Botond, if you have sometime would you please update the patch using slf4j in the testcase.
        Hide
        Botond Hejj added a comment -

        I've attached a patch for 3.3 branch.

        I use log4j because I want to test whether a line is written to the logs or not. I use for that a custom writeappender as I've seen in another testcase. I don't think I could do the same with slf4j.

        Show
        Botond Hejj added a comment - I've attached a patch for 3.3 branch. I use log4j because I want to test whether a line is written to the logs or not. I use for that a custom writeappender as I've seen in another testcase. I don't think I could do the same with slf4j.
        Hide
        Patrick Hunt added a comment -

        I noticed something that needs to be addressed:

        1) the most recent patch replaces slf4j with log4j, which I don't think we want (we want to move trunk to slf4j fully)

        2) there is a prior comment "My last patch was for 3.3 branch" but I can't seem to find it. Could you please attach the 3.3 patch under a separate file name, so that it will be easy to distinguish? Typically we name patches "ZOOKEEPER-####.patch" however in the case of needing more than one patch (for various branches) we typically indicate the non-trunk patch as something like "ZOOKEEPER-####_branch3.3.patch" or similar.

        Thanks!

        Show
        Patrick Hunt added a comment - I noticed something that needs to be addressed: 1) the most recent patch replaces slf4j with log4j, which I don't think we want (we want to move trunk to slf4j fully) 2) there is a prior comment "My last patch was for 3.3 branch" but I can't seem to find it. Could you please attach the 3.3 patch under a separate file name, so that it will be easy to distinguish? Typically we name patches "ZOOKEEPER-####.patch" however in the case of needing more than one patch (for various branches) we typically indicate the non-trunk patch as something like "ZOOKEEPER-####_branch3.3.patch" or similar. Thanks!
        Hide
        Camille Fournier added a comment -

        I looked at this and think it's good, will do another pass on it soon and check it in if it's ready. It would be great if you could give me a patch for the 3.3.X branch though so we can get it there as well.

        Show
        Camille Fournier added a comment - I looked at this and think it's good, will do another pass on it soon and check it in if it's ready. It would be great if you could give me a patch for the 3.3.X branch though so we can get it there as well.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12526375/ZOOKEEPER-1448.patch
        against trunk revision 1336467.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1071//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1071//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1071//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526375/ZOOKEEPER-1448.patch against trunk revision 1336467. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1071//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1071//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1071//console This message is automatically generated.
        Hide
        Botond Hejj added a comment -

        Fixed the test. Actually not an exception was thrown just an ERROR logged in this case with a "should not happen" comment

        Show
        Botond Hejj added a comment - Fixed the test. Actually not an exception was thrown just an ERROR logged in this case with a "should not happen" comment
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12526341/ZOOKEEPER-1448.patch
        against trunk revision 1336467.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The patch appears to cause tar ant target to fail.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1069//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1069//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1069//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526341/ZOOKEEPER-1448.patch against trunk revision 1336467. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1069//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1069//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1069//console This message is automatically generated.
        Hide
        Botond Hejj added a comment -

        My last patch was for 3.3 branch. I've made one for the trunk as well

        Show
        Botond Hejj added a comment - My last patch was for 3.3 branch. I've made one for the trunk as well
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525994/ZOOKEEPER-1448.patch
        against trunk revision 1336467.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1068//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525994/ZOOKEEPER-1448.patch against trunk revision 1336467. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1068//console This message is automatically generated.
        Hide
        Patrick Hunt added a comment -

        Is this ready for review? If so please click "submit patch" button. Thanks.

        Show
        Patrick Hunt added a comment - Is this ready for review? If so please click "submit patch" button. Thanks.
        Hide
        Botond Hejj added a comment -

        I've added a test to DataTreeTest.java. Tell me if that is not the right place.

        Show
        Botond Hejj added a comment - I've added a test to DataTreeTest.java. Tell me if that is not the right place.
        Hide
        Michi Mutsuzaki added a comment -

        Hi Botond,

        Please let us know if you need help writing a test for this. I'm not familiar with this code path myself, but I'm pretty sure some other folks will be able to help.

        Thanks!
        --Michi

        Show
        Michi Mutsuzaki added a comment - Hi Botond, Please let us know if you need help writing a test for this. I'm not familiar with this code path myself, but I'm pretty sure some other folks will be able to help. Thanks! --Michi
        Hide
        Patrick Hunt added a comment -

        Please update the patch to add a test verifying the fix. Thx.

        Show
        Patrick Hunt added a comment - Please update the patch to add a test verifying the fix. Thx.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12523193/ZOOKEEPER-1448.patch
        against trunk revision 1326029.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1041//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1041//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1041//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523193/ZOOKEEPER-1448.patch against trunk revision 1326029. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1041//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1041//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1041//console This message is automatically generated.
        Hide
        Botond Hejj added a comment -

        Patch to clear pTrie before DataTree deserialize

        Show
        Botond Hejj added a comment - Patch to clear pTrie before DataTree deserialize
        Hide
        Camille Fournier added a comment -

        Good catch. Can you provide a patch for this?

        Show
        Camille Fournier added a comment - Good catch. Can you provide a patch for this?

          People

          • Assignee:
            Flavio Junqueira
            Reporter:
            Botond Hejj
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development