[#HADOOP-3565] JavaSerialization can throw java.io.StreamCorruptedException

Hadoop Common

JavaSerialization can throw java.io.StreamCorruptedException

Created: 16/Jun/08 10:59 AM Updated: 08/Jul/09 04:52 PM

Return to search

Component/s:

None

Affects Version/s:

0.17.0, 0.18.0

Fix Version/s:

0.17.1

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
hadoop-3565-v2.patch	2008-06-16 03:16 PM	Tom White	6 kB
hadoop-3565-v3.patch	2008-06-17 09:46 AM	Tom White	6 kB
hadoop-3565.patch	2008-06-16 11:08 AM	Tom White	6 kB

Issue Links:

Blocker

This issue blocks:

MAPREDUCE-252 Create an InputFormat for reading lines of text as Java Strings

Hadoop Flags:	Reviewed, Incompatible change
Release Note:	Change the Java serialization framework, which is not enabled by default, to correctly make the objects independent of the previous objects.
Resolution Date:	17/Jun/08 06:54 PM

Description

« Hide

When using JavaSerialization I sometimes get:

Caused by: java.io.StreamCorruptedException: illegal handle value
        at java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1406)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1285)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
        at org.apache.hadoop.io.serializer.JavaSerialization$JavaSerializationDeserializer.deseriali
ze(JavaSerialization.java:53)
        at org.apache.hadoop.io.serializer.JavaSerialization$JavaSerializationDeserializer.deseriali
ze(JavaSerialization.java:36)
        at org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:735)
        at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:675)
        ... 4 more

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

[ Permlink | « Hide ]

Tom White added a comment - 16/Jun/08 11:08 AM

This happens because java.io.ObjectOutputStream can write back-references to instances previously serialized in the stream, and after the MapReduce shuffle the references can appear before the instances they point to under some circumstances.

Here's a patch to fix this, which writes unshared objects, and also resets the stream to clear shared classes. I've written a unit test too, which fails without this fix.

[ Permlink | « Hide ]

Hadoop QA added a comment - 16/Jun/08 02:50 PM

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12384048/hadoop-3565.patch
against trunk revision 667706.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2662/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2662/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2662/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2662/console

This message is automatically generated.

[ Permlink | « Hide ]

Tom White added a comment - 16/Jun/08 03:16 PM

New patch to fix failing unit test.

[ Permlink | « Hide ]

Hadoop QA added a comment - 16/Jun/08 04:58 PM

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12384063/hadoop-3565-v2.patch
against trunk revision 667706.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2663/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2663/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2663/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2663/console

This message is automatically generated.

[ Permlink | « Hide ]

Tom White added a comment - 17/Jun/08 09:46 AM

Another patch which is compatible with files written using 0.17.0. I tested this manually by writing SequenceFiles using 0.17.0 and reading them back using code with this patch.