Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: ipc
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      A few minor changes are required to get some common classes to work correctly with Avro 1.3 reflection.

      1. HADOOP-6486.patch
        11 kB
        Doug Cutting
      2. HADOOP-6486.patch
        8 kB
        Doug Cutting
      3. HADOOP-6486.patch
        6 kB
        Doug Cutting
      4. HADOOP-6486.patch
        6 kB
        Doug Cutting
      5. HADOOP-6486.patch
        5 kB
        Doug Cutting
      6. HADOOP-6486.patch
        5 kB
        Doug Cutting
      7. HADOOP-6486.patch
        5 kB
        Doug Cutting
      8. HADOOP-6486.patch
        3 kB
        Doug Cutting

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #274 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/274/)
          . Update a few straggling references to Avro 1.2.0 to 1.3.0.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #274 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/274/ ) . Update a few straggling references to Avro 1.2.0 to 1.3.0.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #199 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/199/)
          . Update a few straggling references to Avro 1.2.0 to 1.3.0.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #199 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/199/ ) . Update a few straggling references to Avro 1.2.0 to 1.3.0.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #273 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/273/)
          . fix common classes to work with Avro 1.3 reflection. Contributed by cutting.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #273 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/273/ ) . fix common classes to work with Avro 1.3 reflection. Contributed by cutting.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #197 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/197/)
          . fix common classes to work with Avro 1.3 reflection. Contributed by cutting.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #197 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/197/ ) . fix common classes to work with Avro 1.3 reflection. Contributed by cutting.
          Hide
          Tom White added a comment -

          I've just committed this. Thanks Doug!

          Show
          Tom White added a comment - I've just committed this. Thanks Doug!
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437926/HADOOP-6486.patch
          against trunk revision 918880.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 12 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/31/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/31/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/31/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/31/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437926/HADOOP-6486.patch against trunk revision 918880. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/31/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/31/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/31/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/31/console This message is automatically generated.
          Hide
          Doug Cutting added a comment -

          Forgot to add new test file.

          Show
          Doug Cutting added a comment - Forgot to add new test file.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437919/HADOOP-6486.patch
          against trunk revision 918880.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/409/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/409/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/409/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/409/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437919/HADOOP-6486.patch against trunk revision 918880. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/409/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/409/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/409/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/409/console This message is automatically generated.
          Hide
          Doug Cutting added a comment -

          Here's a new version that adds tests.

          Show
          Doug Cutting added a comment - Here's a new version that adds tests.
          Hide
          Doug Cutting added a comment -

          > Why is EnumSetWritable now a collection?

          Type erasure. Avro reflect knows how to infer a schema for any Collection<Foo>, but not for an arbitrary Wrapper<Foo>. Consider, for example, the following code:

          ...
          private static class Wrapper<T> {
          T wrapped;
          }
          private Wrapper<String> wrapper;
          ...
          

          Java reflection does not make it easy for one to infer a schema for 'wrapper'. The reflected type of that field is a ParametrizedType: Wrapper.class parameterized by String.class. So one might then, to finding the schema for Wrapper.class, pass down String.class as a parameter, use Class#getTypeParameters() to bind T to String.class, then, when T is referenced in the type of the field "wrapped", lookup that binding. I worry about scope issues, since the name T is scoped lexically, while walking the class structure is not, but perhaps that's not an issue. So perhaps Avro reflect could correctly determine a schema for "wrapper", but, at present, it does not.

          Also, representing EnumSetWritable as an array of enum values rather than a one-field-class that wraps an array of enum values seems like a better schema.

          Show
          Doug Cutting added a comment - > Why is EnumSetWritable now a collection? Type erasure. Avro reflect knows how to infer a schema for any Collection<Foo>, but not for an arbitrary Wrapper<Foo>. Consider, for example, the following code: ... private static class Wrapper<T> { T wrapped; } private Wrapper< String > wrapper; ... Java reflection does not make it easy for one to infer a schema for 'wrapper'. The reflected type of that field is a ParametrizedType: Wrapper.class parameterized by String.class. So one might then, to finding the schema for Wrapper.class, pass down String.class as a parameter, use Class#getTypeParameters() to bind T to String.class, then, when T is referenced in the type of the field "wrapped", lookup that binding. I worry about scope issues, since the name T is scoped lexically, while walking the class structure is not, but perhaps that's not an issue. So perhaps Avro reflect could correctly determine a schema for "wrapper", but, at present, it does not. Also, representing EnumSetWritable as an array of enum values rather than a one-field-class that wraps an array of enum values seems like a better schema.
          Hide
          Tom White added a comment -

          +1 Looks good.

          Why is EnumSetWritable now a collection?

          Show
          Tom White added a comment - +1 Looks good. Why is EnumSetWritable now a collection?
          Hide
          Doug Cutting added a comment -

          No new tests are required, since this mostly just updates Hadoop Common to use Avro 1.3.0. The @Stringable annotation is already tested by Avro's unit tests: its use on Path and Text should not need extra tests here.

          Show
          Doug Cutting added a comment - No new tests are required, since this mostly just updates Hadoop Common to use Avro 1.3.0. The @Stringable annotation is already tested by Avro's unit tests: its use on Path and Text should not need extra tests here.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437807/HADOOP-6486.patch
          against trunk revision 918624.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/406/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/406/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/406/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/406/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437807/HADOOP-6486.patch against trunk revision 918624. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/406/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/406/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/406/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/406/console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437804/HADOOP-6486.patch
          against trunk revision 918624.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/405/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/405/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/405/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437804/HADOOP-6486.patch against trunk revision 918624. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/405/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/405/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/405/console This message is automatically generated.
          Hide
          Doug Cutting added a comment -

          New patch to update call to deprecated constructor.

          Show
          Doug Cutting added a comment - New patch to update call to deprecated constructor.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437780/HADOOP-6486.patch
          against trunk revision 918624.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 1025 javac compiler warnings (more than the trunk's current 1024 warnings).

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/403/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/403/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/403/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/403/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437780/HADOOP-6486.patch against trunk revision 918624. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 1025 javac compiler warnings (more than the trunk's current 1024 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/403/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/403/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/403/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/403/console This message is automatically generated.
          Hide
          Doug Cutting added a comment -

          New version that removes Avro dependencies in libraries.properties that now are found through Avro's POM.

          Show
          Doug Cutting added a comment - New version that removes Avro dependencies in libraries.properties that now are found through Avro's POM.
          Hide
          Doug Cutting added a comment -

          Fix ivysettings.xml so default resolver is correctly used by default.

          Show
          Doug Cutting added a comment - Fix ivysettings.xml so default resolver is correctly used by default.
          Hide
          Doug Cutting added a comment -

          Updated version of patch, now that Avro 1.3 is nearly released.

          Note that this changes the ivy configuration so that, as with HDFS and MapReduce, one may develop Common using unreleased versions of other modules (like Avro) by passing -Dresolvers=internal to Ant, so that locally published Maven artifacts are found.

          Show
          Doug Cutting added a comment - Updated version of patch, now that Avro 1.3 is nearly released. Note that this changes the ivy configuration so that, as with HDFS and MapReduce, one may develop Common using unreleased versions of other modules (like Avro) by passing -Dresolvers=internal to Ant, so that locally published Maven artifacts are found.
          Hide
          Doug Cutting added a comment -

          Here's a patch for this. This is not ready to commit since it depends on features that are currently only in Avro's trunk. This should be committed after Avro 1.3 is released.

          Show
          Doug Cutting added a comment - Here's a patch for this. This is not ready to commit since it depends on features that are currently only in Avro's trunk. This should be committed after Avro 1.3 is released.

            People

            • Assignee:
              Doug Cutting
              Reporter:
              Doug Cutting
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development