Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 1.0.2, 2.0.0-alpha
-
Fix Version/s: 1.0.3, 2.6.1, 2.8.0, 2.7.2, 3.0.0-alpha1
-
Labels:
-
Hadoop Flags:Reviewed
Description
SnappyDecompressor.c has the following code in a few places:
THROW(env, "Ljava/lang/InternalError", "Could not decompress data. Buffer length is too small.");
this is incorrect, though, since the THROW macro doesn't need the "L" before the class name. This results in a ClassNotFoundException for Ljava.lang.InternalError being thrown, instead of the intended exception.
Issue Links
- is duplicated by
-
HADOOP-12033
Reducer task failure with java.lang.NoClassDefFoundError: Ljava/lang/InternalError at org.apache.hadoop.io.compress.snappy.SnappyDecompressor.decompressBytesDirect
-
- Resolved
-
-
HADOOP-9666
SnappyCompressor/SnappyDecompressor JNI code uses wrong Exception class name
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Examination of related files shows Todd's comment is manifestly correct. Attached is a patch for branch-1.0. Please review.
Regarding "InternalError" being a questionable choice of Throwable to throw: I guess the implementor wanted Error rather than Exception, and there aren't that many choices of Error.
In branch-1.0, the ZlibCompressor/Decompressor do not have the incorrect "Ljava" with THROW usage.
+1, patch looks good to me. Please upload a trunk patch as well.
Uploaded patch for trunk. Should also work in 0.23 and 2.0. As Todd noted, includes fix for lz4 too.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12525796/HADOOP-8151.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated 2 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/946//console
This message is automatically generated.
Leaving Jira open pending code review and commit of fix for trunk and branch-2.
+1, trunk patch looks good to me. In the future, we should follow the declared policy of committing to trunk before branch-1.
Rebased patch to trunk. Fixed one more instance of L that the earlier patch had missed or the trunk had gotten since. Will commit once jenkins +1's.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12546191/HADOOP-8151.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverController
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1504//console
This message is automatically generated.
Committed this to trunk, thanks Matt.
Integrated in Hadoop-Common-trunk-Commit #2757 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2757/)
HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
Integrated in Hadoop-Hdfs-trunk-Commit #2820 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2820/)
HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
Integrated in Hadoop-Mapreduce-trunk-Commit #2779 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2779/)
HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)
Result = FAILURE
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
Integrated in Hadoop-Hdfs-trunk #1174 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1174/)
HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
Integrated in Hadoop-Mapreduce-trunk #1205 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1205/)
HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) (Revision 1389006)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389006
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
This is not merged into branch-2 and is still an issue there.
Agree. We also meet the same issue in 2.x releases like discussions in HADOOP-12033. I will reopen this JIRA and merge the trunk patch back to branch-2.
Commit patch to branch-2.
Pulled this into 2.6.1 after Akira Ajisaka verified that the patch applies cleanly. Ran compilation before the push.
Just pulled this into branch-2.7 (release 2.7.2) as it already exists in 2.6.1.
branch-2 patch applies cleanly. Ran compilation before the push.
This bug seems to occur in lz4 as well.
It also seems like the wrong kind of exception to throw - InternalError is for JVM-internal unexpected conditions.