Hadoop Common
  1. Hadoop Common
  2. HADOOP-8151

Error handling in snappy decompressor throws invalid exceptions

        Details

        • Type: Bug Bug
        • Status: Resolved
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: 1.0.2, 2.0.0-alpha
        • Fix Version/s: 1.0.3, 3.0.0
        • Component/s: io, native
        • Labels:
          None
        • Hadoop Flags:
          Reviewed

          Description

          SnappyDecompressor.c has the following code in a few places:

              THROW(env, "Ljava/lang/InternalError", "Could not decompress data. Buffer length is too small.");

          this is incorrect, though, since the THROW macro doesn't need the "L" before the class name. This results in a ClassNotFoundException for Ljava.lang.InternalError being thrown, instead of the intended exception.

          1. Text File
            HADOOP-8151-branch-1.0.patch
            2 kB
            Matt Foley
          2. Text File
            HADOOP-8151.patch
            4 kB
            Matt Foley
          3. Text File
            HADOOP-8151.patch
            5 kB
            Harsh J

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            Todd Lipcon added a comment -

            This bug seems to occur in lz4 as well.

            It also seems like the wrong kind of exception to throw - InternalError is for JVM-internal unexpected conditions.

            Show
            Todd Lipcon added a comment - This bug seems to occur in lz4 as well. It also seems like the wrong kind of exception to throw - InternalError is for JVM-internal unexpected conditions.
            Hide
            Permalink
            Matt Foley added a comment -

            Examination of related files shows Todd's comment is manifestly correct. Attached is a patch for branch-1.0. Please review.

            Show
            Matt Foley added a comment - Examination of related files shows Todd's comment is manifestly correct. Attached is a patch for branch-1.0. Please review.
            Hide
            Permalink
            Matt Foley added a comment -

            Regarding "InternalError" being a questionable choice of Throwable to throw: I guess the implementor wanted Error rather than Exception, and there aren't that many choices of Error.

            In branch-1.0, the ZlibCompressor/Decompressor do not have the incorrect "Ljava" with THROW usage.

            Show
            Matt Foley added a comment - Regarding "InternalError" being a questionable choice of Throwable to throw: I guess the implementor wanted Error rather than Exception, and there aren't that many choices of Error. In branch-1.0, the ZlibCompressor/Decompressor do not have the incorrect "Ljava" with THROW usage.
            Hide
            Permalink
            Todd Lipcon added a comment -

            +1, patch looks good to me. Please upload a trunk patch as well.

            Show
            Todd Lipcon added a comment - +1, patch looks good to me. Please upload a trunk patch as well.
            Hide
            Permalink
            Matt Foley added a comment -

            Uploaded patch for trunk. Should also work in 0.23 and 2.0. As Todd noted, includes fix for lz4 too.

            Show
            Matt Foley added a comment - Uploaded patch for trunk. Should also work in 0.23 and 2.0. As Todd noted, includes fix for lz4 too.
            Hide
            Permalink
            Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12525796/HADOOP-8151.patch
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//console

            This message is automatically generated.

            Show
            Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525796/HADOOP-8151.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//console This message is automatically generated.
            Hide
            Permalink
            Matt Foley added a comment -

            Leaving Jira open pending code review and commit of fix for trunk and branch-2.

            Show
            Matt Foley added a comment - Leaving Jira open pending code review and commit of fix for trunk and branch-2.
            Hide
            Permalink
            Todd Lipcon added a comment -

            +1, trunk patch looks good to me. In the future, we should follow the declared policy of committing to trunk before branch-1.

            Show
            Todd Lipcon added a comment - +1, trunk patch looks good to me. In the future, we should follow the declared policy of committing to trunk before branch-1.
            Hide
            Permalink
            Harsh J added a comment -

            Rebased patch to trunk. Fixed one more instance of L that the earlier patch had missed or the trunk had gotten since. Will commit once jenkins +1's.

            Show
            Harsh J added a comment - Rebased patch to trunk. Fixed one more instance of L that the earlier patch had missed or the trunk had gotten since. Will commit once jenkins +1's.
            Hide
            Permalink
            Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12546191/HADOOP-8151.patch
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

            org.apache.hadoop.ha.TestZKFailoverController

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//console

            This message is automatically generated.

            Show
            Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546191/HADOOP-8151.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//console This message is automatically generated.
            Hide
            Permalink
            Harsh J added a comment -

            Committed this to trunk, thanks Matt.

            Show
            Harsh J added a comment - Committed this to trunk, thanks Matt.
            Hide
            Permalink
            Hudson added a comment -

            Integrated in Hadoop-Common-trunk-Commit #2757 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2757/)
            HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

            Result = SUCCESS
            harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
            Files :

            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
            Show
            Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2757 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2757/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
            Hide
            Permalink
            Hudson added a comment -

            Integrated in Hadoop-Hdfs-trunk-Commit #2820 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2820/)
            HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

            Result = SUCCESS
            harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
            Files :

            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
            Show
            Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2820 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2820/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
            Hide
            Permalink
            Hudson added a comment -

            Integrated in Hadoop-Mapreduce-trunk-Commit #2779 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2779/)
            HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

            Result = FAILURE
            harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
            Files :

            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
            Show
            Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2779 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2779/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
            Hide
            Permalink
            Hudson added a comment -

            Integrated in Hadoop-Hdfs-trunk #1174 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1174/)
            HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

            Result = SUCCESS
            harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
            Files :

            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
            Show
            Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1174 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1174/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
            Hide
            Permalink
            Hudson added a comment -

            Integrated in Hadoop-Mapreduce-trunk #1205 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1205/)
            HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)

            Result = SUCCESS
            harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
            Files :

            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
            • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
            Show
            Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1205 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1205/ ) HADOOP-8151 . Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
            Hide
            Permalink
            Gopal V added a comment -

            This is not merged into branch-2 and is still an issue there.

            Show
            Gopal V added a comment - This is not merged into branch-2 and is still an issue there.

              People

              • Assignee:
                Matt Foley
                Reporter:
                Todd Lipcon
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development