Hadoop Common
  1. Hadoop Common
  2. HADOOP-8151

Error handling in snappy decompressor throws invalid exceptions

        Details

        • Type: Bug Bug
        • Status: Resolved
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: 1.0.2, 2.0.0-alpha
        • Fix Version/s: 1.0.3, 3.0.0
        • Component/s: io, native
        • Labels:
          None
        • Hadoop Flags:
          Reviewed

          Description

          SnappyDecompressor.c has the following code in a few places:

              THROW(env, "Ljava/lang/InternalError", "Could not decompress data. Buffer length is too small.");

          this is incorrect, though, since the THROW macro doesn't need the "L" before the class name. This results in a ClassNotFoundException for Ljava.lang.InternalError being thrown, instead of the intended exception.

          1. Text File
            HADOOP-8151-branch-1.0.patch
            2 kB
            Matt Foley
          2. Text File
            HADOOP-8151.patch
            4 kB
            Matt Foley
          3. Text File
            HADOOP-8151.patch
            5 kB
            Harsh J

            Issue Links

            is duplicated by

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-9666 SnappyCompressor/SnappyDecompressor JNI code uses wrong Exception class name

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Resolved

              Activity

              Descending order - Click to sort in ascending order
              Hide
              Permalink
              Gopal V added a comment -

              This is not merged into branch-2 and is still an issue there.

              Show
              Gopal V added a comment - This is not merged into branch-2 and is still an issue there.
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Mapreduce-trunk #1205 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1205/)
              HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

              Result = SUCCESS
              harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
              Files :

              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
              Show
              Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1205 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1205/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Hdfs-trunk #1174 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1174/)
              HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

              Result = SUCCESS
              harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
              Files :

              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
              Show
              Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1174 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1174/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Mapreduce-trunk-Commit #2779 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2779/)
              HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

              Result = FAILURE
              harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
              Files :

              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
              Show
              Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2779 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2779/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Hdfs-trunk-Commit #2820 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2820/)
              HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

              Result = SUCCESS
              harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
              Files :

              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
              Show
              Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2820 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2820/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Common-trunk-Commit #2757 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2757/)
              HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

              Result = SUCCESS
              harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
              Files :

              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
              Show
              Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2757 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2757/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
              Hide
              Permalink
              Harsh J added a comment -

              Committed this to trunk, thanks Matt.

              Show
              Harsh J added a comment - Committed this to trunk, thanks Matt.
              Hide
              Permalink
              Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12546191/HADOOP-8151.patch
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              -1 tests included. The patch doesn't appear to include any new or modified tests.
              Please justify why no new tests are needed for this patch.
              Also please list what manual steps were performed to verify this patch.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

              org.apache.hadoop.ha.TestZKFailoverController

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//testReport/
              Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//console

              This message is automatically generated.

              Show
              Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546191/HADOOP-8151.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//console This message is automatically generated.
              Hide
              Permalink
              Harsh J added a comment -

              Rebased patch to trunk. Fixed one more instance of L that the earlier patch had missed or the trunk had gotten since. Will commit once jenkins +1's.

              Show
              Harsh J added a comment - Rebased patch to trunk. Fixed one more instance of L that the earlier patch had missed or the trunk had gotten since. Will commit once jenkins +1's.
              Hide
              Permalink
              Todd Lipcon added a comment -

              +1, trunk patch looks good to me. In the future, we should follow the declared policy of committing to trunk before branch-1.

              Show
              Todd Lipcon added a comment - +1, trunk patch looks good to me. In the future, we should follow the declared policy of committing to trunk before branch-1.
              Hide
              Permalink
              Matt Foley added a comment -

              Leaving Jira open pending code review and commit of fix for trunk and branch-2.

              Show
              Matt Foley added a comment - Leaving Jira open pending code review and commit of fix for trunk and branch-2.
              Hide
              Permalink
              Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12525796/HADOOP-8151.patch
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              -1 tests included. The patch doesn't appear to include any new or modified tests.
              Please justify why no new tests are needed for this patch.
              Also please list what manual steps were performed to verify this patch.

              -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//testReport/
              Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//console

              This message is automatically generated.

              Show
              Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525796/HADOOP-8151.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//console This message is automatically generated.
              Hide
              Permalink
              Matt Foley added a comment -

              Uploaded patch for trunk. Should also work in 0.23 and 2.0. As Todd noted, includes fix for lz4 too.

              Show
              Matt Foley added a comment - Uploaded patch for trunk. Should also work in 0.23 and 2.0. As Todd noted, includes fix for lz4 too.
              Hide
              Permalink
              Todd Lipcon added a comment -

              +1, patch looks good to me. Please upload a trunk patch as well.

              Show
              Todd Lipcon added a comment - +1, patch looks good to me. Please upload a trunk patch as well.
              Hide
              Permalink
              Matt Foley added a comment -

              Regarding "InternalError" being a questionable choice of Throwable to throw: I guess the implementor wanted Error rather than Exception, and there aren't that many choices of Error.

              In branch-1.0, the ZlibCompressor/Decompressor do not have the incorrect "Ljava" with THROW usage.

              Show
              Matt Foley added a comment - Regarding "InternalError" being a questionable choice of Throwable to throw: I guess the implementor wanted Error rather than Exception, and there aren't that many choices of Error. In branch-1.0, the ZlibCompressor/Decompressor do not have the incorrect "Ljava" with THROW usage.
              Hide
              Permalink
              Matt Foley added a comment -

              Examination of related files shows Todd's comment is manifestly correct. Attached is a patch for branch-1.0. Please review.

              Show
              Matt Foley added a comment - Examination of related files shows Todd's comment is manifestly correct. Attached is a patch for branch-1.0. Please review.
              Hide
              Permalink
              Todd Lipcon added a comment -

              This bug seems to occur in lz4 as well.

              It also seems like the wrong kind of exception to throw - InternalError is for JVM-internal unexpected conditions.

              Show
              Todd Lipcon added a comment - This bug seems to occur in lz4 as well. It also seems like the wrong kind of exception to throw - InternalError is for JVM-internal unexpected conditions.

                People

                • Assignee:
                  Matt Foley
                  Reporter:
                  Todd Lipcon
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  9 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development