Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5053

java.lang.InternalError from decompression codec cause reducer to fail

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: trunk, 2.0.3-alpha, 0.23.5
    • Fix Version/s: 3.0.0, 0.23.7, 2.0.4-alpha
    • Component/s: None
    • Labels:
      None

      Description

      lz4, snappy, zlib, and lzo Decompressor's only throw java.lang.InternalError. This exception will cause the reducer to fail and bypass all of the fetch failure logic. The decompressing errors should be treated as fetch failures.

      1. MAPREDUCE-5053-b023-1.patch
        2 kB
        Robert Parker
      2. MAPREDUCE-5053-2.patch
        6 kB
        Robert Parker
      3. MAPREDUCE-5053-1.patch
        6 kB
        Robert Parker

        Activity

        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12572678/MAPREDUCE-5053-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 tests included appear to have a timeout.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3393//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3393//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572678/MAPREDUCE-5053-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3393//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3393//console This message is automatically generated.
        Hide
        Jonathan Eagles added a comment -

        Minor nit. Comment about mock object return null in new test should be updated. Also, if you can provide a 23 patch, I can check it in there also.

        Show
        Jonathan Eagles added a comment - Minor nit. Comment about mock object return null in new test should be updated. Also, if you can provide a 23 patch, I can check it in there also.
        Hide
        Robert Parker added a comment -

        John, I have uploaded a new patch for trunk and branch-0.23. The unit test for branch 0.23 was not possible due to the private and static functions that could not be mocked.

        Show
        Robert Parker added a comment - John, I have uploaded a new patch for trunk and branch-0.23. The unit test for branch 0.23 was not possible due to the private and static functions that could not be mocked.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12572853/MAPREDUCE-5053-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 tests included appear to have a timeout.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3395//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3395//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572853/MAPREDUCE-5053-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3395//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3395//console This message is automatically generated.
        Hide
        Jonathan Eagles added a comment -

        +1. Thanks for the fix, Rob.

        Show
        Jonathan Eagles added a comment - +1. Thanks for the fix, Rob.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3490 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3490/)
        MAPREDUCE-5053. java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458350)

        Result = SUCCESS
        jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458350
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3490 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3490/ ) MAPREDUCE-5053 . java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458350) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458350 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #161 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/161/)
        MAPREDUCE-5053. java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458350)

        Result = SUCCESS
        jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458350
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #161 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/161/ ) MAPREDUCE-5053 . java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458350) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458350 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #559 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/559/)
        MAPREDUCE-5053. java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458368)

        Result = SUCCESS
        jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458368
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #559 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/559/ ) MAPREDUCE-5053 . java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458368) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458368 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1350 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1350/)
        MAPREDUCE-5053. java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458350)

        Result = FAILURE
        jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458350
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1350 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1350/ ) MAPREDUCE-5053 . java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458350) Result = FAILURE jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458350 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1378 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1378/)
        MAPREDUCE-5053. java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458350)

        Result = SUCCESS
        jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458350
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1378 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1378/ ) MAPREDUCE-5053 . java.lang.InternalError from decompression codec cause reducer to fail (Robert Parker via jeagles) (Revision 1458350) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458350 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Siddharth Seth added a comment -

        Jonathan Eagles , Is the fix version correct ? - did this go into 2.0.4-alpha, or to branch-2 (2.0.5-beta)

        Show
        Siddharth Seth added a comment - Jonathan Eagles , Is the fix version correct ? - did this go into 2.0.4-alpha, or to branch-2 (2.0.5-beta)
        Hide
        Sangjin Lee added a comment -

        I do not think this made it into 2.0.4 or 2.0.5. I believe the proper fixed version is 2.1.0 for hadoop 2.

        Show
        Sangjin Lee added a comment - I do not think this made it into 2.0.4 or 2.0.5. I believe the proper fixed version is 2.1.0 for hadoop 2.

          People

          • Assignee:
            Robert Parker
            Reporter:
            Robert Parker
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development