Hadoop Common
  1. Hadoop Common
  2. HADOOP-5450

Add support for application-specific typecodes to typed bytes

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.2, 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      For serializing objects of types that are not supported by typed bytes serialization, applications might want to use a custom serialization format. Right now, typecode 0 has to be used for the bytes resulting from this custom serialization, which could lead to problems when deserializing the objects because the application cannot know if a byte sequence following typecode 0 is a customly serialized object or just a raw sequence of bytes. Therefore, a range of typecodes that are treated as aliases for 0 should be added, such that different typecodes can be used for application-specific purposes.

      1. HADOOP-5450.patch
        10 kB
        Klaas Bosteels

        Activity

        Hide
        Arun C Murthy added a comment -

        Thanks Matt. I've committed to branch-1 after running tests.

        Show
        Arun C Murthy added a comment - Thanks Matt. I've committed to branch-1 after running tests.
        Hide
        Matt Foley added a comment -

        +1. Please push it to both branch-1 and branch-1.0. Thanks.

        Show
        Matt Foley added a comment - +1. Please push it to both branch-1 and branch-1.0. Thanks.
        Hide
        Arun C Murthy added a comment -

        Matt - Klaas's patch applies clean to branch-1. I'd like to push it in for hadoop-1.0.2, you ok? Thanks.

        Show
        Arun C Murthy added a comment - Matt - Klaas's patch applies clean to branch-1. I'd like to push it in for hadoop-1.0.2, you ok? Thanks.
        Hide
        Ravi Gummadi added a comment -

        Looks like "the revert of other unnecessary code changes done" also reverted the needed changes to TestIO.java unintentionally in trunk ?

        Show
        Ravi Gummadi added a comment - Looks like "the revert of other unnecessary code changes done" also reverted the needed changes to TestIO.java unintentionally in trunk ?
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #797 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/797/)
        . Revert accidental inclusion.
        . Add application-specific data types to streaming's typed bytes
        interface. (Klaas Bosteels via omalley)

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #797 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/797/ ) . Revert accidental inclusion. . Add application-specific data types to streaming's typed bytes interface. (Klaas Bosteels via omalley)
        Hide
        Owen O'Malley added a comment -

        I just committed this. Thanks, Klaas!

        Show
        Owen O'Malley added a comment - I just committed this. Thanks, Klaas!
        Hide
        Klaas Bosteels added a comment -

        This should really go into 0.21, so making it a blocker for that release.

        Show
        Klaas Bosteels added a comment - This should really go into 0.21, so making it a blocker for that release.
        Hide
        Klaas Bosteels added a comment -

        The failed unit tests are not related to the patch.

        Show
        Klaas Bosteels added a comment - The failed unit tests are not related to the patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12401846/HADOOP-5450.patch
        against trunk revision 752292.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/46/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/46/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/46/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/46/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12401846/HADOOP-5450.patch against trunk revision 752292. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/46/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/46/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/46/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/46/console This message is automatically generated.
        Hide
        Klaas Bosteels added a comment -

        The attached patch makes sure that the typecode range 50-200 can be used for application-specific purposes. As an example, it also makes TypedBytesWritableInput and TypedBytesWritableOutput use typecode 50 for serializing Writables that do not match with any of the types supported by the typed bytes format.

        Show
        Klaas Bosteels added a comment - The attached patch makes sure that the typecode range 50-200 can be used for application-specific purposes. As an example, it also makes TypedBytesWritableInput and TypedBytesWritableOutput use typecode 50 for serializing Writables that do not match with any of the types supported by the typed bytes format.

          People

          • Assignee:
            Klaas Bosteels
            Reporter:
            Klaas Bosteels
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development