Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5053

java.lang.InternalError from decompression codec cause reducer to fail

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha, 0.23.5
    • Fix Version/s: 0.23.7, 2.0.4-alpha
    • Component/s: None
    • Labels:
      None

      Description

      lz4, snappy, zlib, and lzo Decompressor's only throw java.lang.InternalError. This exception will cause the reducer to fail and bypass all of the fetch failure logic. The decompressing errors should be treated as fetch failures.

      1. MAPREDUCE-5053-2.patch
        6 kB
        Robert Parker
      2. MAPREDUCE-5053-b023-1.patch
        2 kB
        Robert Parker
      3. MAPREDUCE-5053-1.patch
        6 kB
        Robert Parker

        Issue Links

          Activity

          Hide
          Sangjin Lee added a comment -

          I do not think this made it into 2.0.4 or 2.0.5. I believe the proper fixed version is 2.1.0 for hadoop 2.

          Show
          Sangjin Lee added a comment - I do not think this made it into 2.0.4 or 2.0.5. I believe the proper fixed version is 2.1.0 for hadoop 2.
          Hide
          Siddharth Seth added a comment -

          Jonathan Eagles , Is the fix version correct ? - did this go into 2.0.4-alpha, or to branch-2 (2.0.5-beta)

          Show
          Siddharth Seth added a comment - Jonathan Eagles , Is the fix version correct ? - did this go into 2.0.4-alpha, or to branch-2 (2.0.5-beta)
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1378 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1378/)
          MAPREDUCE-5053. java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458350)

          Result = SUCCESS
          jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458350
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1378 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1378/ ) MAPREDUCE-5053 . java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458350) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458350 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1350 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1350/)
          MAPREDUCE-5053. java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458350)

          Result = FAILURE
          jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458350
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1350 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1350/ ) MAPREDUCE-5053 . java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458350) Result = FAILURE jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458350 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #559 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/559/)
          MAPREDUCE-5053. java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458368)

          Result = SUCCESS
          jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458368
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #559 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/559/ ) MAPREDUCE-5053 . java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458368) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458368 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #161 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/161/)
          MAPREDUCE-5053. java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458350)

          Result = SUCCESS
          jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458350
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #161 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/161/ ) MAPREDUCE-5053 . java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458350) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458350 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3490 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3490/)
          MAPREDUCE-5053. java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458350)

          Result = SUCCESS
          jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458350
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3490 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3490/ ) MAPREDUCE-5053 . java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458350) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458350 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          Hide
          Jonathan Eagles added a comment -

          +1. Thanks for the fix, Rob.

          Show
          Jonathan Eagles added a comment - +1. Thanks for the fix, Rob.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12572853/MAPREDUCE-5053-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3395//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3395//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572853/MAPREDUCE-5053-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3395//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3395//console This message is automatically generated.
          Hide
          Robert Parker added a comment -

          John, I have uploaded a new patch for trunk and branch-0.23. The unit test for branch 0.23 was not possible due to the private and static functions that could not be mocked.

          Show
          Robert Parker added a comment - John, I have uploaded a new patch for trunk and branch-0.23. The unit test for branch 0.23 was not possible due to the private and static functions that could not be mocked.
          Hide
          Jonathan Eagles added a comment -

          Minor nit. Comment about mock object return null in new test should be updated. Also, if you can provide a 23 patch, I can check it in there also.

          Show
          Jonathan Eagles added a comment - Minor nit. Comment about mock object return null in new test should be updated. Also, if you can provide a 23 patch, I can check it in there also.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12572678/MAPREDUCE-5053-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3393//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3393//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572678/MAPREDUCE-5053-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3393//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3393//console This message is automatically generated.

            People

            • Assignee:
              Robert Parker
              Reporter:
              Robert Parker
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development