Hadoop Common
  1. Hadoop Common
  2. HADOOP-8151

Error handling in snappy decompressor throws invalid exceptions

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.2, 2.0.0-alpha
    • Fix Version/s: 1.0.3, 3.0.0
    • Component/s: io, native
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      SnappyDecompressor.c has the following code in a few places:

          THROW(env, "Ljava/lang/InternalError", "Could not decompress data. Buffer length is too small.");
      

      this is incorrect, though, since the THROW macro doesn't need the "L" before the class name. This results in a ClassNotFoundException for Ljava.lang.InternalError being thrown, instead of the intended exception.

      1. HADOOP-8151-branch-1.0.patch
        2 kB
        Matt Foley
      2. HADOOP-8151.patch
        4 kB
        Matt Foley
      3. HADOOP-8151.patch
        5 kB
        Harsh J

        Issue Links

          Activity

          Hide
          Gopal V added a comment -

          This is not merged into branch-2 and is still an issue there.

          Show
          Gopal V added a comment - This is not merged into branch-2 and is still an issue there.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1205 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1205/)
          HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1205 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1205/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1174 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1174/)
          HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1174 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1174/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2779 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2779/)
          HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

          Result = FAILURE
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2779 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2779/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2820 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2820/)
          HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2820 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2820/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2757 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2757/)
          HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2757 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2757/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
          Hide
          Harsh J added a comment -

          Committed this to trunk, thanks Matt.

          Show
          Harsh J added a comment - Committed this to trunk, thanks Matt.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12546191/HADOOP-8151.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.ha.TestZKFailoverController

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546191/HADOOP-8151.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Rebased patch to trunk. Fixed one more instance of L that the earlier patch had missed or the trunk had gotten since. Will commit once jenkins +1's.

          Show
          Harsh J added a comment - Rebased patch to trunk. Fixed one more instance of L that the earlier patch had missed or the trunk had gotten since. Will commit once jenkins +1's.
          Hide
          Todd Lipcon added a comment -

          +1, trunk patch looks good to me. In the future, we should follow the declared policy of committing to trunk before branch-1.

          Show
          Todd Lipcon added a comment - +1, trunk patch looks good to me. In the future, we should follow the declared policy of committing to trunk before branch-1.
          Hide
          Matt Foley added a comment -

          Leaving Jira open pending code review and commit of fix for trunk and branch-2.

          Show
          Matt Foley added a comment - Leaving Jira open pending code review and commit of fix for trunk and branch-2.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12525796/HADOOP-8151.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525796/HADOOP-8151.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//console This message is automatically generated.
          Hide
          Matt Foley added a comment -

          Uploaded patch for trunk. Should also work in 0.23 and 2.0. As Todd noted, includes fix for lz4 too.

          Show
          Matt Foley added a comment - Uploaded patch for trunk. Should also work in 0.23 and 2.0. As Todd noted, includes fix for lz4 too.
          Hide
          Todd Lipcon added a comment -

          +1, patch looks good to me. Please upload a trunk patch as well.

          Show
          Todd Lipcon added a comment - +1, patch looks good to me. Please upload a trunk patch as well.
          Hide
          Matt Foley added a comment -

          Regarding "InternalError" being a questionable choice of Throwable to throw: I guess the implementor wanted Error rather than Exception, and there aren't that many choices of Error.

          In branch-1.0, the ZlibCompressor/Decompressor do not have the incorrect "Ljava" with THROW usage.

          Show
          Matt Foley added a comment - Regarding "InternalError" being a questionable choice of Throwable to throw: I guess the implementor wanted Error rather than Exception, and there aren't that many choices of Error. In branch-1.0, the ZlibCompressor/Decompressor do not have the incorrect "Ljava" with THROW usage.
          Hide
          Matt Foley added a comment -

          Examination of related files shows Todd's comment is manifestly correct. Attached is a patch for branch-1.0. Please review.

          Show
          Matt Foley added a comment - Examination of related files shows Todd's comment is manifestly correct. Attached is a patch for branch-1.0. Please review.
          Hide
          Todd Lipcon added a comment -

          This bug seems to occur in lz4 as well.

          It also seems like the wrong kind of exception to throw - InternalError is for JVM-internal unexpected conditions.

          Show
          Todd Lipcon added a comment - This bug seems to occur in lz4 as well. It also seems like the wrong kind of exception to throw - InternalError is for JVM-internal unexpected conditions.

            People

            • Assignee:
              Matt Foley
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development