Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13270

BZip2CompressionInputStream finds the same compression marker twice in corner case, causing duplicate data blocks

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Unit test TestTextInputFormat.testSplitableCodecs() failed when the seed is 1313094493.

      Stacktrace
      java.lang.AssertionError: Key in multiple partitions.
      at org.junit.Assert.fail(Assert.java:88)
      at org.junit.Assert.assertTrue(Assert.java:41)
      at org.junit.Assert.assertFalse(Assert.java:64)
      at org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs(TestTextInputFormat.java:223)

      1. MAPREDUCE-6643.05.patch
        6 kB
        Kai Sasaki
      2. MAPREDUCE-6643.04.patch
        6 kB
        Kai Sasaki
      3. MAPREDUCE-6643.03.patch
        6 kB
        Kai Sasaki
      4. MAPREDUCE-6643.02.patch
        7 kB
        Kai Sasaki
      5. MAPREDUCE-6643.01.patch
        7 kB
        Kai Sasaki

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 28s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 7m 17s trunk passed
          +1 compile 7m 38s trunk passed
          +1 checkstyle 1m 32s trunk passed
          +1 mvnsite 1m 26s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 1m 55s trunk passed
          +1 javadoc 1m 6s trunk passed
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 1m 4s the patch passed
          +1 compile 7m 40s the patch passed
          +1 javac 7m 40s the patch passed
          +1 checkstyle 1m 32s root: The patch generated 0 new + 53 unchanged - 2 fixed = 53 total (was 55)
          +1 mvnsite 1m 30s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 2m 14s the patch passed
          +1 javadoc 1m 1s the patch passed
          -1 unit 16m 59s hadoop-common in the patch failed.
          +1 unit 114m 33s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 asflicense 0m 26s The patch does not generate ASF License warnings.
          170m 55s



          Reason Tests
          Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809833/MAPREDUCE-6643.05.patch
          JIRA Issue HADOOP-13270
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5408869a6cab 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 709a814
          Default Java 1.8.0_91
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9764/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9764/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9764/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9764/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 7m 17s trunk passed +1 compile 7m 38s trunk passed +1 checkstyle 1m 32s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 1m 55s trunk passed +1 javadoc 1m 6s trunk passed 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 4s the patch passed +1 compile 7m 40s the patch passed +1 javac 7m 40s the patch passed +1 checkstyle 1m 32s root: The patch generated 0 new + 53 unchanged - 2 fixed = 53 total (was 55) +1 mvnsite 1m 30s the patch passed +1 mvneclipse 0m 27s the patch passed -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 2m 14s the patch passed +1 javadoc 1m 1s the patch passed -1 unit 16m 59s hadoop-common in the patch failed. +1 unit 114m 33s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 170m 55s Reason Tests Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809833/MAPREDUCE-6643.05.patch JIRA Issue HADOOP-13270 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5408869a6cab 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 709a814 Default Java 1.8.0_91 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9764/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9764/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9764/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9764/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to branch-2.7 and above. Thanks Kai for the contribution and thanks Haibo for the review.

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to branch-2.7 and above. Thanks Kai for the contribution and thanks Haibo for the review.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Moved to Hadoop common and updated the summary because the change is in hadoop-common.

          Show
          ajisakaa Akira Ajisaka added a comment - Moved to Hadoop common and updated the summary because the change is in hadoop-common.
          Hide
          haibochen Haibo Chen added a comment -

          Yep, the latest patch fixes all failure cases (seeds) I have.

          Show
          haibochen Haibo Chen added a comment - Yep, the latest patch fixes all failure cases (seeds) I have.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1. Hi Haibo Chen, would you test the latest patch? I'll commit it if you say okay.

          Show
          ajisakaa Akira Ajisaka added a comment - +1. Hi Haibo Chen , would you test the latest patch? I'll commit it if you say okay.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 32s Maven dependency ordering for branch
          +1 mvninstall 6m 21s trunk passed
          +1 compile 6m 34s trunk passed
          +1 checkstyle 1m 19s trunk passed
          +1 mvnsite 1m 19s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 55s trunk passed
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 0m 57s the patch passed
          +1 compile 6m 20s the patch passed
          +1 javac 6m 20s the patch passed
          +1 checkstyle 1m 19s root: The patch generated 0 new + 53 unchanged - 2 fixed = 53 total (was 55)
          +1 mvnsite 1m 16s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 1m 57s the patch passed
          +1 javadoc 0m 56s the patch passed
          +1 unit 7m 32s hadoop-common in the patch passed.
          -1 unit 108m 59s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          150m 34s



          Reason Tests
          Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809833/MAPREDUCE-6643.05.patch
          JIRA Issue MAPREDUCE-6643
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c616dd15fbfb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 28b66ae
          Default Java 1.8.0_91
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6551/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6551/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6551/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6551/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6551/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 32s Maven dependency ordering for branch +1 mvninstall 6m 21s trunk passed +1 compile 6m 34s trunk passed +1 checkstyle 1m 19s trunk passed +1 mvnsite 1m 19s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 55s trunk passed 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 0m 57s the patch passed +1 compile 6m 20s the patch passed +1 javac 6m 20s the patch passed +1 checkstyle 1m 19s root: The patch generated 0 new + 53 unchanged - 2 fixed = 53 total (was 55) +1 mvnsite 1m 16s the patch passed +1 mvneclipse 0m 26s the patch passed -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 1m 57s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 7m 32s hadoop-common in the patch passed. -1 unit 108m 59s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 150m 34s Reason Tests Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809833/MAPREDUCE-6643.05.patch JIRA Issue MAPREDUCE-6643 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c616dd15fbfb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 28b66ae Default Java 1.8.0_91 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6551/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6551/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6551/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6551/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6551/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          lewuathe Kai Sasaki added a comment -

          whitespace issue seems to be caused by this issue reported in HADOOP-13258.
          https://issues.apache.org/jira/browse/HADOOP-13258

          Show
          lewuathe Kai Sasaki added a comment - whitespace issue seems to be caused by this issue reported in HADOOP-13258 . https://issues.apache.org/jira/browse/HADOOP-13258
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 6m 11s trunk passed
          +1 compile 6m 28s trunk passed
          +1 checkstyle 1m 20s trunk passed
          +1 mvnsite 1m 19s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 1m 42s trunk passed
          +1 javadoc 0m 58s trunk passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 0m 57s the patch passed
          +1 compile 6m 23s the patch passed
          +1 javac 6m 23s the patch passed
          +1 checkstyle 1m 19s root: The patch generated 0 new + 54 unchanged - 2 fixed = 54 total (was 56)
          +1 mvnsite 1m 18s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 1m 56s the patch passed
          +1 javadoc 0m 57s the patch passed
          +1 unit 6m 59s hadoop-common in the patch passed.
          -1 unit 104m 42s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          145m 20s



          Reason Tests
          Failed junit tests hadoop.mapreduce.TestMRJobClient
            hadoop.mapred.TestMRCJCFileOutputCommitter



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809791/MAPREDUCE-6643.04.patch
          JIRA Issue MAPREDUCE-6643
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ae197954b98c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 28b66ae
          Default Java 1.8.0_91
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6550/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6550/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6550/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6550/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6550/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 6m 11s trunk passed +1 compile 6m 28s trunk passed +1 checkstyle 1m 20s trunk passed +1 mvnsite 1m 19s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 58s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 0m 57s the patch passed +1 compile 6m 23s the patch passed +1 javac 6m 23s the patch passed +1 checkstyle 1m 19s root: The patch generated 0 new + 54 unchanged - 2 fixed = 54 total (was 56) +1 mvnsite 1m 18s the patch passed +1 mvneclipse 0m 26s the patch passed -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 1m 56s the patch passed +1 javadoc 0m 57s the patch passed +1 unit 6m 59s hadoop-common in the patch passed. -1 unit 104m 42s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 145m 20s Reason Tests Failed junit tests hadoop.mapreduce.TestMRJobClient   hadoop.mapred.TestMRCJCFileOutputCommitter Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809791/MAPREDUCE-6643.04.patch JIRA Issue MAPREDUCE-6643 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ae197954b98c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 28b66ae Default Java 1.8.0_91 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6550/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6550/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6550/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6550/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6550/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          lewuathe Kai Sasaki added a comment -

          Akira Ajisaka Thanks you for reviewing. I updated the patch accordingly.
          And also thanks for checking test case Haibo Chen.

          Show
          lewuathe Kai Sasaki added a comment - Akira Ajisaka Thanks you for reviewing. I updated the patch accordingly. And also thanks for checking test case Haibo Chen .
          Hide
          haibochen Haibo Chen added a comment -

          Tested the patch against some other seeds with which I have seen the test failed. All green after applying the patch.

          Show
          haibochen Haibo Chen added a comment - Tested the patch against some other seeds with which I have seen the test failed. All green after applying the patch.
          Hide
          ajisakaa Akira Ajisaka added a comment - - edited

          Thanks Kai Sasaki for updating the patch! Mostly looks good to me. Some comments:

              if (start == 0) {
                adjStart = Math.max(0L, start - FIRST_BZIP2_BLOCK_MARKER_POSITION);
          

          1. It seems that adjStart is always zero. If that is true, we should set zero instead of using max function.

                // Other than the fist of file, the marker size is 6 bytes.
          

          2. fist -> first
          3. Would you fix the checkstyle warning?

              // corner case when we have byte alignment and position of stream are same
              int[] lengths = {471507, 473608};
              int[] numSplits = {218, 110};
          
              for (int i = 0; i < 2; i++) {
                verifyPartitions(lengths[i], numSplits[i], file, codec, conf);
              }
          

          4. (nit) The following code is simpler for me:

              // corner case when we have byte alignment and position of stream are same
              verifyPartitions(471507, 218, file, codec, conf);
              verifyPartitions(473608, 110, file, codec, conf);
          
          Show
          ajisakaa Akira Ajisaka added a comment - - edited Thanks Kai Sasaki for updating the patch! Mostly looks good to me. Some comments: if (start == 0) { adjStart = Math .max(0L, start - FIRST_BZIP2_BLOCK_MARKER_POSITION); 1. It seems that adjStart is always zero. If that is true, we should set zero instead of using max function. // Other than the fist of file, the marker size is 6 bytes. 2. fist -> first 3. Would you fix the checkstyle warning? // corner case when we have byte alignment and position of stream are same int [] lengths = {471507, 473608}; int [] numSplits = {218, 110}; for ( int i = 0; i < 2; i++) { verifyPartitions(lengths[i], numSplits[i], file, codec, conf); } 4. (nit) The following code is simpler for me: // corner case when we have byte alignment and position of stream are same verifyPartitions(471507, 218, file, codec, conf); verifyPartitions(473608, 110, file, codec, conf);
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 12s Maven dependency ordering for branch
          +1 mvninstall 6m 23s trunk passed
          +1 compile 6m 41s trunk passed
          +1 checkstyle 1m 22s trunk passed
          +1 mvnsite 1m 23s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 1m 0s trunk passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 1m 0s the patch passed
          +1 compile 6m 41s the patch passed
          +1 javac 6m 41s the patch passed
          -1 checkstyle 1m 20s root: The patch generated 1 new + 54 unchanged - 2 fixed = 55 total (was 56)
          +1 mvnsite 1m 21s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 2m 0s the patch passed
          +1 javadoc 0m 58s the patch passed
          -1 unit 21m 30s hadoop-common in the patch failed.
          -1 unit 109m 37s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          165m 53s



          Reason Tests
          Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809696/MAPREDUCE-6643.03.patch
          JIRA Issue MAPREDUCE-6643
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 72dc0063f6ed 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5279af7
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 6m 23s trunk passed +1 compile 6m 41s trunk passed +1 checkstyle 1m 22s trunk passed +1 mvnsite 1m 23s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 1m 0s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 0s the patch passed +1 compile 6m 41s the patch passed +1 javac 6m 41s the patch passed -1 checkstyle 1m 20s root: The patch generated 1 new + 54 unchanged - 2 fixed = 55 total (was 56) +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 26s the patch passed -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 2m 0s the patch passed +1 javadoc 0m 58s the patch passed -1 unit 21m 30s hadoop-common in the patch failed. -1 unit 109m 37s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 165m 53s Reason Tests Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809696/MAPREDUCE-6643.03.patch JIRA Issue MAPREDUCE-6643 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 72dc0063f6ed 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5279af7 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          lewuathe Kai Sasaki added a comment -

          I found BZip2Codec always back 10bytes to search compression marker even markers except for first marker has 6bytes. It can let BZip2CompressionInputStream find same marker twice because the search start points is back more than marker size. I fixed compression stream only back 6bytes other than the file start point.

          Show
          lewuathe Kai Sasaki added a comment - I found BZip2Codec always back 10bytes to search compression marker even markers except for first marker has 6bytes. It can let BZip2CompressionInputStream find same marker twice because the search start points is back more than marker size. I fixed compression stream only back 6bytes other than the file start point.
          Hide
          lewuathe Kai Sasaki added a comment -

          Akira Ajisaka Thanks you so much for checking related JIRA. As you said, I found the current patch will cause same problem fixed in HADOOP-11445 although a patch in HADOOP-11445 also seems cause this problem.
          So looks like we need to investigate further and solve root cause. I'll look into and update the patch.

          Show
          lewuathe Kai Sasaki added a comment - Akira Ajisaka Thanks you so much for checking related JIRA. As you said, I found the current patch will cause same problem fixed in HADOOP-11445 although a patch in HADOOP-11445 also seems cause this problem. So looks like we need to investigate further and solve root cause. I'll look into and update the patch.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Haibo Chen and Andras Bokor for reporting this and thanks Kai Sasaki for providing the patch.
          Hi Kai Sasaki, this patch looks to be reverting HADOOP-11445, so the change would cause HADOOP-11445 again. What do you think?

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Haibo Chen and Andras Bokor for reporting this and thanks Kai Sasaki for providing the patch. Hi Kai Sasaki , this patch looks to be reverting HADOOP-11445 , so the change would cause HADOOP-11445 again. What do you think?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 22s Maven dependency ordering for branch
          +1 mvninstall 6m 0s trunk passed
          +1 compile 6m 10s trunk passed
          +1 checkstyle 1m 17s trunk passed
          +1 mvnsite 1m 15s trunk passed
          +1 mvneclipse 0m 23s trunk passed
          +1 findbugs 1m 37s trunk passed
          +1 javadoc 1m 4s trunk passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 0m 56s the patch passed
          +1 compile 6m 11s the patch passed
          +1 javac 6m 11s the patch passed
          +1 checkstyle 1m 18s root: The patch generated 0 new + 52 unchanged - 4 fixed = 52 total (was 56)
          +1 mvnsite 1m 16s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 52s the patch passed
          +1 javadoc 1m 5s the patch passed
          -1 unit 6m 31s hadoop-common in the patch failed.
          -1 unit 109m 59s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          149m 29s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS
            hadoop.mapred.TestMRCJCFileOutputCommitter
            hadoop.mapred.TestMiniMRChildTask



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808141/MAPREDUCE-6643.02.patch
          JIRA Issue MAPREDUCE-6643
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux dbedf73cb935 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 106234d
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6536/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6536/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6536/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6536/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6536/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6536/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 22s Maven dependency ordering for branch +1 mvninstall 6m 0s trunk passed +1 compile 6m 10s trunk passed +1 checkstyle 1m 17s trunk passed +1 mvnsite 1m 15s trunk passed +1 mvneclipse 0m 23s trunk passed +1 findbugs 1m 37s trunk passed +1 javadoc 1m 4s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 0m 56s the patch passed +1 compile 6m 11s the patch passed +1 javac 6m 11s the patch passed +1 checkstyle 1m 18s root: The patch generated 0 new + 52 unchanged - 4 fixed = 52 total (was 56) +1 mvnsite 1m 16s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 52s the patch passed +1 javadoc 1m 5s the patch passed -1 unit 6m 31s hadoop-common in the patch failed. -1 unit 109m 59s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 149m 29s Reason Tests Failed junit tests hadoop.net.TestDNS   hadoop.mapred.TestMRCJCFileOutputCommitter   hadoop.mapred.TestMiniMRChildTask Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808141/MAPREDUCE-6643.02.patch JIRA Issue MAPREDUCE-6643 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dbedf73cb935 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 106234d Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6536/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6536/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6536/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6536/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6536/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6536/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 28s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 6m 1s trunk passed
          +1 compile 6m 11s trunk passed
          +1 checkstyle 1m 19s trunk passed
          +1 mvnsite 1m 16s trunk passed
          +1 mvneclipse 0m 23s trunk passed
          +1 findbugs 1m 35s trunk passed
          +1 javadoc 1m 5s trunk passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 0m 55s the patch passed
          +1 compile 6m 9s the patch passed
          +1 javac 6m 9s the patch passed
          -1 checkstyle 1m 19s root: The patch generated 4 new + 52 unchanged - 4 fixed = 56 total (was 56)
          +1 mvnsite 1m 16s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 52s the patch passed
          +1 javadoc 1m 3s the patch passed
          +1 unit 7m 4s hadoop-common in the patch passed.
          -1 unit 108m 45s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          148m 42s



          Reason Tests
          Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter
            hadoop.mapred.TestMiniMRChildTask



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808121/MAPREDUCE-6643.01.patch
          JIRA Issue MAPREDUCE-6643
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 086dab996252 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 106234d
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6535/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6535/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6535/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6535/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6535/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 6m 1s trunk passed +1 compile 6m 11s trunk passed +1 checkstyle 1m 19s trunk passed +1 mvnsite 1m 16s trunk passed +1 mvneclipse 0m 23s trunk passed +1 findbugs 1m 35s trunk passed +1 javadoc 1m 5s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 0m 55s the patch passed +1 compile 6m 9s the patch passed +1 javac 6m 9s the patch passed -1 checkstyle 1m 19s root: The patch generated 4 new + 52 unchanged - 4 fixed = 56 total (was 56) +1 mvnsite 1m 16s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 52s the patch passed +1 javadoc 1m 3s the patch passed +1 unit 7m 4s hadoop-common in the patch passed. -1 unit 108m 45s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 148m 42s Reason Tests Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter   hadoop.mapred.TestMiniMRChildTask Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808121/MAPREDUCE-6643.01.patch JIRA Issue MAPREDUCE-6643 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 086dab996252 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 106234d Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6535/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6535/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6535/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6535/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6535/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          lewuathe Kai Sasaki added a comment -

          It seems to be caused by non checking corner case when byte alignment of data and current position of stream is same. In this case BZip2 stream can return same block marker same to previous one.
          I fixed BZip2Codec and added above test cases.

          Show
          lewuathe Kai Sasaki added a comment - It seems to be caused by non checking corner case when byte alignment of data and current position of stream is same. In this case BZip2 stream can return same block marker same to previous one. I fixed BZip2Codec and added above test cases.
          Hide
          boky01 Andras Bokor added a comment -

          When seed is 855631073 it also fails.

          Show
          boky01 Andras Bokor added a comment - When seed is 855631073 it also fails.

            People

            • Assignee:
              lewuathe Kai Sasaki
              Reporter:
              haibochen Haibo Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development