[#HADOOP-5450] Add support for application-specific typecodes to typed bytes

Hadoop Common

Add support for application-specific typecodes to typed bytes

Created: 10/Mar/09 09:13 AM Updated: 08/Jul/09 05:05 PM

Component/s:

None

Affects Version/s:

None

Fix Version/s:

0.21.0

Time Tracking:

Not Specified

File Attachments:

	File Name	Date Attached	Attached By	Size
	HADOOP-5450.patch	2009-03-10 03:27 PM	Klaas Bosteels	10 kB

Hadoop Flags:	Reviewed
Resolution Date:	03/Apr/09 11:23 PM
Labels:

Description

« Hide

For serializing objects of types that are not supported by typed bytes serialization, applications might want to use a custom serialization format. Right now, typecode 0 has to be used for the bytes resulting from this custom serialization, which could lead to problems when deserializing the objects because the application cannot know if a byte sequence following typecode 0 is a customly serialized object or just a raw sequence of bytes. Therefore, a range of typecodes that are treated as aliases for 0 should be added, such that different typecodes can be used for application-specific purposes.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

[ Permalink | « Hide ]

Klaas Bosteels added a comment - 10/Mar/09 03:27 PM

The attached patch makes sure that the typecode range 50-200 can be used for application-specific purposes. As an example, it also makes TypedBytesWritableInput and TypedBytesWritableOutput use typecode 50 for serializing Writables that do not match with any of the types supported by the typed bytes format.

Klaas Bosteels made changes - 10/Mar/09 03:27 PM

Field	Original Value	New Value
Attachment		HADOOP-5450.patch [ 12401846 ]

Klaas Bosteels made changes - 10/Mar/09 03:28 PM

Status

Open [ 1 ]

Patch Available [ 10002 ]

[ Permalink | « Hide ]

Hadoop QA added a comment - 11/Mar/09 04:15 AM

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12401846/HADOOP-5450.patch
against trunk revision 752292.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/46/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/46/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/46/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/46/console

This message is automatically generated.

[ Permalink | « Hide ]

Klaas Bosteels added a comment - 11/Mar/09 08:18 AM

The failed unit tests are not related to the patch.

[ Permalink | « Hide ]

Klaas Bosteels added a comment - 11/Mar/09 01:36 PM

This should really go into 0.21, so making it a blocker for that release.

Klaas Bosteels made changes - 11/Mar/09 01:36 PM

Fix Version/s		0.21.0 [ 12313563 ]
Priority	Major [ 3 ]	Blocker [ 1 ]

Repository	Revision	Date	User	Message
ASF	#761837	Fri Apr 03 23:17:27 UTC 2009	omalley	~~HADOOP-5450~~. Add application-specific data types to streaming's typed bytes interface. (Klaas Bosteels via omalley)
				Files Changed
				MODIFY /hadoop/core/trunk/src/contrib/streaming/src/java/org/apache/hadoop/typedbytes/Type.java MODIFY /hadoop/core/trunk/src/hdfs/org/apache/hadoop/hdfs/HftpFileSystem.java MODIFY /hadoop/core/trunk/src/mapred/org/apache/hadoop/mapred/Task.java MODIFY /hadoop/core/trunk/src/core/org/apache/hadoop/fs/RawLocalFileSystem.java MODIFY /hadoop/core/trunk/src/contrib/streaming/src/java/org/apache/hadoop/typedbytes/package.html MODIFY /hadoop/core/trunk/src/core/org/apache/hadoop/fs/ftp/FTPFileSystem.java MODIFY /hadoop/core/trunk/src/hdfs/org/apache/hadoop/hdfs/DistributedFileSystem.java MODIFY /hadoop/core/trunk/src/core/org/apache/hadoop/fs/FileSystem.java MODIFY /hadoop/core/trunk/src/core/org/apache/hadoop/fs/s3/S3FileSystem.java MODIFY /hadoop/core/trunk/src/core/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java MODIFY /hadoop/core/trunk/src/contrib/streaming/src/test/org/apache/hadoop/typedbytes/TestIO.java MODIFY /hadoop/core/trunk/src/core/org/apache/hadoop/fs/kfs/KosmosFileSystem.java MODIFY /hadoop/core/trunk/CHANGES.txt MODIFY /hadoop/core/trunk/src/contrib/streaming/src/java/org/apache/hadoop/typedbytes/TypedBytesWritableOutput.java MODIFY /hadoop/core/trunk/src/contrib/streaming/src/java/org/apache/hadoop/typedbytes/TypedBytesOutput.java MODIFY /hadoop/core/trunk/src/mapred/org/apache/hadoop/mapred/Child.java MODIFY /hadoop/core/trunk/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java MODIFY /hadoop/core/trunk/src/contrib/streaming/src/java/org/apache/hadoop/typedbytes/TypedBytesWritableInput.java MODIFY /hadoop/core/trunk/src/contrib/streaming/src/java/org/apache/hadoop/typedbytes/TypedBytesInput.java

Repository	Revision	Date	User	Message
ASF	#761838	Fri Apr 03 23:21:46 UTC 2009	omalley	~~HADOOP-5450~~. Revert accidental inclusion.
				Files Changed
				MODIFY /hadoop/core/trunk/src/contrib/streaming/src/test/org/apache/hadoop/typedbytes/TestIO.java MODIFY /hadoop/core/trunk/src/core/org/apache/hadoop/fs/kfs/KosmosFileSystem.java MODIFY /hadoop/core/trunk/src/mapred/org/apache/hadoop/mapred/Task.java MODIFY /hadoop/core/trunk/src/hdfs/org/apache/hadoop/hdfs/HftpFileSystem.java MODIFY /hadoop/core/trunk/src/core/org/apache/hadoop/fs/ftp/FTPFileSystem.java MODIFY /hadoop/core/trunk/src/core/org/apache/hadoop/fs/RawLocalFileSystem.java MODIFY /hadoop/core/trunk/src/hdfs/org/apache/hadoop/hdfs/DistributedFileSystem.java MODIFY /hadoop/core/trunk/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java MODIFY /hadoop/core/trunk/src/mapred/org/apache/hadoop/mapred/Child.java MODIFY /hadoop/core/trunk/src/core/org/apache/hadoop/fs/FileSystem.java MODIFY /hadoop/core/trunk/src/core/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java MODIFY /hadoop/core/trunk/src/core/org/apache/hadoop/fs/s3/S3FileSystem.java

[ Permalink | « Hide ]

Owen O'Malley added a comment - 03/Apr/09 11:23 PM

I just committed this. Thanks, Klaas!

Owen O'Malley made changes - 03/Apr/09 11:23 PM

Status	Patch Available [ 10002 ]	Resolved [ 5 ]
Hadoop Flags		[Reviewed]
Resolution		Fixed [ 1 ]

[ Permalink | « Hide ]

Hudson added a comment - 04/Apr/09 03:15 PM

Integrated in Hadoop-trunk #797 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/797/)
. Revert accidental inclusion.
. Add application-specific data types to streaming's typed bytes
interface. (Klaas Bosteels via omalley)

Owen O'Malley made changes - 08/Jul/09 05:05 PM

Component/s

contrib/streaming [ 12310972 ]