Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9939

Increase DecompressorStream skip buffer size

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None

      Description

      See ACCUMULO-2353 for details.

      Filing this jira to investigate performance difference and possibly make the buf size change accordingly.

      1. HDFS-9939.001.patch
        18 kB
        John Zhuge
      2. HDFS-9939.002.patch
        19 kB
        John Zhuge

        Issue Links

          Activity

          Hide
          jzhuge John Zhuge added a comment -

          Patch 001:

          • Set skip buffer size to 2048 (MAX_SKIP_BUFFER_SIZE)
          • Move FakeCompressor and FakeDecompressor out of TestBlockDecompressorStream.java into their own files
          • Add TestDecompressorStream with test cases for read and skip
          Show
          jzhuge John Zhuge added a comment - Patch 001: Set skip buffer size to 2048 (MAX_SKIP_BUFFER_SIZE) Move FakeCompressor and FakeDecompressor out of TestBlockDecompressorStream.java into their own files Add TestDecompressorStream with test cases for read and skip
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 7m 29s trunk passed
          +1 compile 8m 46s trunk passed with JDK v1.8.0_74
          +1 compile 7m 47s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 1m 5s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 45s trunk passed
          +1 javadoc 1m 4s trunk passed with JDK v1.8.0_74
          +1 javadoc 1m 14s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 48s the patch passed
          +1 compile 8m 42s the patch passed with JDK v1.8.0_74
          +1 javac 8m 42s the patch passed
          +1 compile 7m 54s the patch passed with JDK v1.7.0_95
          +1 javac 7m 54s the patch passed
          +1 checkstyle 0m 22s hadoop-common-project/hadoop-common: patch generated 0 new + 5 unchanged - 1 fixed = 5 total (was 6)
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 56s the patch passed
          +1 javadoc 1m 2s the patch passed with JDK v1.8.0_74
          +1 javadoc 1m 15s the patch passed with JDK v1.7.0_95
          -1 unit 9m 49s hadoop-common in the patch failed with JDK v1.8.0_74.
          -1 unit 9m 16s hadoop-common in the patch failed with JDK v1.7.0_95.
          -1 asflicense 0m 23s Patch generated 2 ASF License warnings.
          74m 2s



          Reason Tests
          JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795146/HDFS-9939.001.patch
          JIRA Issue HDFS-9939
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 98ececdb8b11 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 19b645c
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14919/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14919/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14919/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14919/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14919/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14919/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14919/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 7m 29s trunk passed +1 compile 8m 46s trunk passed with JDK v1.8.0_74 +1 compile 7m 47s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 1m 5s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 1m 4s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 14s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 48s the patch passed +1 compile 8m 42s the patch passed with JDK v1.8.0_74 +1 javac 8m 42s the patch passed +1 compile 7m 54s the patch passed with JDK v1.7.0_95 +1 javac 7m 54s the patch passed +1 checkstyle 0m 22s hadoop-common-project/hadoop-common: patch generated 0 new + 5 unchanged - 1 fixed = 5 total (was 6) +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 56s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_74 +1 javadoc 1m 15s the patch passed with JDK v1.7.0_95 -1 unit 9m 49s hadoop-common in the patch failed with JDK v1.8.0_74. -1 unit 9m 16s hadoop-common in the patch failed with JDK v1.7.0_95. -1 asflicense 0m 23s Patch generated 2 ASF License warnings. 74m 2s Reason Tests JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795146/HDFS-9939.001.patch JIRA Issue HDFS-9939 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 98ececdb8b11 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 19b645c Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14919/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14919/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14919/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14919/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14919/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14919/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14919/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Only one little nit, unused variable buf in testSkip. Otherwise LGTM, thanks John!

          Show
          andrew.wang Andrew Wang added a comment - Only one little nit, unused variable buf in testSkip. Otherwise LGTM, thanks John!
          Hide
          jzhuge John Zhuge added a comment -

          Thanks. That could have been caught by HADOOP-12701 since I ran test-patch.

          John Zhuge

          Show
          jzhuge John Zhuge added a comment - Thanks. That could have been caught by HADOOP-12701 since I ran test-patch. John Zhuge
          Hide
          jzhuge John Zhuge added a comment -

          Patch 002:

          • Remove unused variable buf in testSkip

          Andrew Wang, could you please review and commit?

          Show
          jzhuge John Zhuge added a comment - Patch 002: Remove unused variable buf in testSkip Andrew Wang , could you please review and commit?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 6m 51s trunk passed
          +1 compile 6m 1s trunk passed with JDK v1.8.0_91
          +1 compile 6m 47s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 36s trunk passed
          +1 javadoc 0m 58s trunk passed with JDK v1.8.0_91
          +1 javadoc 1m 6s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 41s the patch passed
          +1 compile 5m 53s the patch passed with JDK v1.8.0_91
          +1 javac 5m 53s the patch passed
          +1 compile 6m 44s the patch passed with JDK v1.7.0_95
          +1 javac 6m 44s the patch passed
          +1 checkstyle 0m 20s hadoop-common-project/hadoop-common: patch generated 0 new + 5 unchanged - 1 fixed = 5 total (was 6)
          +1 mvnsite 0m 56s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 50s the patch passed
          +1 javadoc 0m 54s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 6s the patch passed with JDK v1.7.0_95
          -1 unit 19m 15s hadoop-common in the patch failed with JDK v1.8.0_91.
          +1 unit 7m 58s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          72m 34s



          Reason Tests
          JDK v1.8.0_91 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802560/HDFS-9939.002.patch
          JIRA Issue HDFS-9939
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2a1f50955b9b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f305d9c
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15377/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15377/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15377/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15377/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 6m 51s trunk passed +1 compile 6m 1s trunk passed with JDK v1.8.0_91 +1 compile 6m 47s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 36s trunk passed +1 javadoc 0m 58s trunk passed with JDK v1.8.0_91 +1 javadoc 1m 6s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 41s the patch passed +1 compile 5m 53s the patch passed with JDK v1.8.0_91 +1 javac 5m 53s the patch passed +1 compile 6m 44s the patch passed with JDK v1.7.0_95 +1 javac 6m 44s the patch passed +1 checkstyle 0m 20s hadoop-common-project/hadoop-common: patch generated 0 new + 5 unchanged - 1 fixed = 5 total (was 6) +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 50s the patch passed +1 javadoc 0m 54s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 6s the patch passed with JDK v1.7.0_95 -1 unit 19m 15s hadoop-common in the patch failed with JDK v1.8.0_91. +1 unit 7m 58s hadoop-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 72m 34s Reason Tests JDK v1.8.0_91 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802560/HDFS-9939.002.patch JIRA Issue HDFS-9939 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2a1f50955b9b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f305d9c Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15377/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15377/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15377/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15377/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Committed to trunk, branch-2, branch-2.8, thanks for the contribution John!

          Show
          andrew.wang Andrew Wang added a comment - Committed to trunk, branch-2, branch-2.8, thanks for the contribution John!
          Hide
          jzhuge John Zhuge added a comment -
          Show
          jzhuge John Zhuge added a comment - Thanks Andrew Wang and Yongjun Zhang !
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9732 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9732/)
          HDFS-9939. Increase DecompressorStream skip buffer size. Contributed by (wang: rev 8ef75a702e3b4b8ca7794421c44da7f8d44daa4c)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/FakeDecompressor.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/FakeCompressor.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/DecompressorStream.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestBlockDecompressorStream.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestDecompressorStream.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9732 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9732/ ) HDFS-9939 . Increase DecompressorStream skip buffer size. Contributed by (wang: rev 8ef75a702e3b4b8ca7794421c44da7f8d44daa4c) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/FakeDecompressor.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/FakeCompressor.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/DecompressorStream.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestBlockDecompressorStream.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestDecompressorStream.java

            People

            • Assignee:
              jzhuge John Zhuge
              Reporter:
              yzhangal Yongjun Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development