Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-3496

Transaction larger than jute.maxbuffer makes ZooKeeper unavailable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 3.5.5, 3.4.14
    • 3.6.0, 3.5.7
    • None

    Description

      Problem:
      ZooKeeper server fails to start, logs following error

      Exception in thread "main" java.io.IOException: Unreasonable length = 1001025
               at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
               at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
      

      This indicates that one of the transactions size is more than the configured jute.maxbuffer values. But how transaction more than jute.maxbuffer size is allowed to write?

      Analysis:
      At ZooKeeper server jute.maxbuffer specifies the maximum size of a transaction. By default it is 1 MB at the server
      jute.maxbuffer is used for following:

      1. Size sanity check of incoming request. Incoming requests size must not be more than jute.maxbuffer
      2. Size sanity check of the transaction while reading from transaction or snapshot file. Transaction size must not be more than jute.maxbuffer+1024
      3. Size sanity check of transaction while reading data from the leader. Transaction size must not be more than jute.maxbuffer+1024

      Request size sanity check is done in the beginning of a request processing but later request processing adds additional information into request then writes to transaction file. This additional information size is not considered in sanity check. This is how transaction larger than jute.maxbuffer are accepted into ZooKeeper.

      If this additional information size is less than 1024 Bytes then it is OK as ZooKeeper already takes care of it.
      But if this additional information size is more than 1024 bytes it allows the request, But while reading from transaction/snapshot file and while reading from leader it fails and make the ZooKeeper service unavailable

      Example:
      Suppose incoming request size is 1000000 Bytes
      Configured jute.maxbuffer is 1000000
      After processing the request ZooKeeper server adds 1025 more bytes
      In this case, request will be processed successfully, and 1000000+1025 bytes will be written to transaction file
      But while reading from the transaction log 1000000+1025 bytes cannot be read as max allowed length is 1000000(effectively 1000000+1024).

      Solutions:
      If incoming request size sanity check is done after populating all additional information then this problem is solved. But doing sanity check in the later stage of request processing will defeat the purpose of sanity check itself. So this we can not do

      Currently additional information size is constant 1024 Bytes Code Reference. We should increase this value and make it more reasonable. I propose to make this additional information size to same as the jute.maxbuffer. Also make additional information size configurable.

      Attachments

        1. ZOOKEEPER-3496-001.patch
          8 kB
          Mohammad Arshad

        Issue Links

          Activity

            This problem is same as ZOOKEEPER-2101. Created new JIRA to start fresh discussion.

            arshad.mohammad Mohammad Arshad added a comment - This problem is same as ZOOKEEPER-2101 . Created new JIRA to start fresh discussion.

            Attached Fixed ZOOKEEPER-3496-001.patch for reference, will create PR later

            arshad.mohammad Mohammad Arshad added a comment - Attached Fixed ZOOKEEPER-3496 -001.patch for reference, will create PR later
            lvfangmin Fangmin Lv added a comment -

            arshad.mohammad thanks for reporting the issue and providing the patch, it looks good to me, do you mind to create a PR for this.

            lvfangmin Fangmin Lv added a comment - arshad.mohammad  thanks for reporting the issue and providing the patch, it looks good to me, do you mind to create a PR for this.

            PR Created

            arshad.mohammad Mohammad Arshad added a comment - PR Created
            hadoopqa Hadoop QA added a comment -

            +1 overall. GitHub Pull Request Build

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4148//testReport/
            Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4148//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4148//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4148//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4148//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4148//console This message is automatically generated.
            hadoopqa Hadoop QA added a comment -

            +1 overall. GitHub Pull Request Build

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4150//testReport/
            Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4150//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4150//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4150//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4150//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4150//console This message is automatically generated.
            hadoopqa Hadoop QA added a comment -

            +1 overall. GitHub Pull Request Build

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4152//testReport/
            Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4152//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4152//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4152//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4152//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4152//console This message is automatically generated.

            Committed to master branch
            Thank you arshad.mohammad

            eolivelli Enrico Olivelli added a comment - Committed to master branch Thank you arshad.mohammad

            hanm had committed to branch-3.5. This issue is committed to branch-3.5 and master

            arshad.mohammad Mohammad Arshad added a comment - hanm had committed to branch-3.5. This issue is committed to branch-3.5 and master

            Thanks eolivelli, hanm, lvfangmin for the reviews.

            arshad.mohammad Mohammad Arshad added a comment - Thanks eolivelli , hanm , lvfangmin for the reviews.
            hudson Hudson added a comment -

            SUCCESS: Integrated in Jenkins build Zookeeper-trunk-single-thread #548 (See https://builds.apache.org/job/Zookeeper-trunk-single-thread/548/)
            ZOOKEEPER-3496: Transaction larger than jute.maxbuffer makes ZooKeeper (enrico.olivelli: rev 4279758ead655aff34cdff21c9f2c71d66030d14)

            • (edit) zookeeper-jute/src/test/java/org/apache/jute/BinaryInputArchiveTest.java
            • (edit) zookeeper-jute/src/main/java/org/apache/jute/BinaryInputArchive.java
            • (edit) zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
            hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Zookeeper-trunk-single-thread #548 (See https://builds.apache.org/job/Zookeeper-trunk-single-thread/548/ ) ZOOKEEPER-3496 : Transaction larger than jute.maxbuffer makes ZooKeeper (enrico.olivelli: rev 4279758ead655aff34cdff21c9f2c71d66030d14) (edit) zookeeper-jute/src/test/java/org/apache/jute/BinaryInputArchiveTest.java (edit) zookeeper-jute/src/main/java/org/apache/jute/BinaryInputArchive.java (edit) zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
            hudson Hudson added a comment -

            SUCCESS: Integrated in Jenkins build ZooKeeper-trunk #711 (See https://builds.apache.org/job/ZooKeeper-trunk/711/)
            ZOOKEEPER-3496: Transaction larger than jute.maxbuffer makes ZooKeeper (enrico.olivelli: rev 4279758ead655aff34cdff21c9f2c71d66030d14)

            • (edit) zookeeper-jute/src/test/java/org/apache/jute/BinaryInputArchiveTest.java
            • (edit) zookeeper-jute/src/main/java/org/apache/jute/BinaryInputArchive.java
            • (edit) zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
            hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build ZooKeeper-trunk #711 (See https://builds.apache.org/job/ZooKeeper-trunk/711/ ) ZOOKEEPER-3496 : Transaction larger than jute.maxbuffer makes ZooKeeper (enrico.olivelli: rev 4279758ead655aff34cdff21c9f2c71d66030d14) (edit) zookeeper-jute/src/test/java/org/apache/jute/BinaryInputArchiveTest.java (edit) zookeeper-jute/src/main/java/org/apache/jute/BinaryInputArchive.java (edit) zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md

            People

              arshad.mohammad Mohammad Arshad
              arshad.mohammad Mohammad Arshad
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 6h 20m
                  6h 20m