Hadoop Common
  1. Hadoop Common
  2. HADOOP-8151

Error handling in snappy decompressor throws invalid exceptions

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.2, 2.0.0-alpha
    • Fix Version/s: 1.0.3, 3.0.0
    • Component/s: io, native
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      SnappyDecompressor.c has the following code in a few places:

          THROW(env, "Ljava/lang/InternalError", "Could not decompress data. Buffer length is too small.");
      

      this is incorrect, though, since the THROW macro doesn't need the "L" before the class name. This results in a ClassNotFoundException for Ljava.lang.InternalError being thrown, instead of the intended exception.

      1. HADOOP-8151-branch-1.0.patch
        2 kB
        Matt Foley
      2. HADOOP-8151.patch
        4 kB
        Matt Foley
      3. HADOOP-8151.patch
        5 kB
        Harsh J

        Activity

        Hide
        Todd Lipcon added a comment -

        This bug seems to occur in lz4 as well.

        It also seems like the wrong kind of exception to throw - InternalError is for JVM-internal unexpected conditions.

        Show
        Todd Lipcon added a comment - This bug seems to occur in lz4 as well. It also seems like the wrong kind of exception to throw - InternalError is for JVM-internal unexpected conditions.
        Hide
        Matt Foley added a comment -

        Examination of related files shows Todd's comment is manifestly correct. Attached is a patch for branch-1.0. Please review.

        Show
        Matt Foley added a comment - Examination of related files shows Todd's comment is manifestly correct. Attached is a patch for branch-1.0. Please review.
        Hide
        Matt Foley added a comment -

        Regarding "InternalError" being a questionable choice of Throwable to throw: I guess the implementor wanted Error rather than Exception, and there aren't that many choices of Error.

        In branch-1.0, the ZlibCompressor/Decompressor do not have the incorrect "Ljava" with THROW usage.

        Show
        Matt Foley added a comment - Regarding "InternalError" being a questionable choice of Throwable to throw: I guess the implementor wanted Error rather than Exception, and there aren't that many choices of Error. In branch-1.0, the ZlibCompressor/Decompressor do not have the incorrect "Ljava" with THROW usage.
        Hide
        Todd Lipcon added a comment -

        +1, patch looks good to me. Please upload a trunk patch as well.

        Show
        Todd Lipcon added a comment - +1, patch looks good to me. Please upload a trunk patch as well.
        Hide
        Matt Foley added a comment -

        Uploaded patch for trunk. Should also work in 0.23 and 2.0. As Todd noted, includes fix for lz4 too.

        Show
        Matt Foley added a comment - Uploaded patch for trunk. Should also work in 0.23 and 2.0. As Todd noted, includes fix for lz4 too.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525796/HADOOP-8151.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525796/HADOOP-8151.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//console This message is automatically generated.
        Hide
        Matt Foley added a comment -

        Leaving Jira open pending code review and commit of fix for trunk and branch-2.

        Show
        Matt Foley added a comment - Leaving Jira open pending code review and commit of fix for trunk and branch-2.
        Hide
        Todd Lipcon added a comment -

        +1, trunk patch looks good to me. In the future, we should follow the declared policy of committing to trunk before branch-1.

        Show
        Todd Lipcon added a comment - +1, trunk patch looks good to me. In the future, we should follow the declared policy of committing to trunk before branch-1.
        Hide
        Harsh J added a comment -

        Rebased patch to trunk. Fixed one more instance of L that the earlier patch had missed or the trunk had gotten since. Will commit once jenkins +1's.

        Show
        Harsh J added a comment - Rebased patch to trunk. Fixed one more instance of L that the earlier patch had missed or the trunk had gotten since. Will commit once jenkins +1's.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12546191/HADOOP-8151.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.ha.TestZKFailoverController

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546191/HADOOP-8151.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//console This message is automatically generated.
        Hide
        Harsh J added a comment -

        Committed this to trunk, thanks Matt.

        Show
        Harsh J added a comment - Committed this to trunk, thanks Matt.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2757 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2757/)
        HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

        Result = SUCCESS
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2757 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2757/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2820 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2820/)
        HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

        Result = SUCCESS
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2820 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2820/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2779 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2779/)
        HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

        Result = FAILURE
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2779 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2779/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1174 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1174/)
        HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

        Result = SUCCESS
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1174 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1174/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1205 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1205/)
        HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

        Result = SUCCESS
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1205 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1205/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
        Hide
        Gopal V added a comment -

        This is not merged into branch-2 and is still an issue there.

        Show
        Gopal V added a comment - This is not merged into branch-2 and is still an issue there.

          People

          • Assignee:
            Matt Foley
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development