Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.23.0
    • Component/s: io
    • Labels:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Tags:
      avro version

      Description

      Needed by mapreduce 2.0 avro support. Maybe we could jump to Avro 1.5. There is incompatible API changes from 1.3x to 1.4x (Utf8 to CharSequence in user facing APIs) not sure about 1.5x though.

      1. hadoop-7264-mr2-v1.patch
        9 kB
        Luke Lu
      2. hadoop-7264-v1.patch
        30 kB
        Luke Lu
      3. HADOOP-7264v2.patch
        12 kB
        Alejandro Abdelnur
      4. HADOOP-7264v3.patch
        11 kB
        Alejandro Abdelnur
      5. HADOOP-7264v4.patch
        11 kB
        Alejandro Abdelnur
      6. HADOOP-7264v5.patch
        37 kB
        Alejandro Abdelnur
      7. HADOOP-7264v6.patch
        16 kB
        Alejandro Abdelnur

        Issue Links

          Activity

          Hide
          Alejandro Abdelnur added a comment -

          (enter too fast), where are you seeing a failure?

          Show
          Alejandro Abdelnur added a comment - (enter too fast), where are you seeing a failure?
          Hide
          Alejandro Abdelnur added a comment -

          @Luke, I have not seen any failure related to this

          Show
          Alejandro Abdelnur added a comment - @Luke, I have not seen any failure related to this
          Hide
          Luke Lu added a comment -

          Not sure about the current status of HDFS mavenization. But the trunk HDFS build might fail because paranamer-ant transitive dependency is gone. HDFS-2268 was part of my original v1 patch.

          Show
          Luke Lu added a comment - Not sure about the current status of HDFS mavenization. But the trunk HDFS build might fail because paranamer-ant transitive dependency is gone. HDFS-2268 was part of my original v1 patch.
          Hide
          Tom White added a comment -

          I've just committed this. Thanks Alejandro and Luke!

          Show
          Tom White added a comment - I've just committed this. Thanks Alejandro and Luke!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #757 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/757/)
          HADOOP-7264. Bump avro version to at least 1.4.1. Contributed by Alejandro Abdelnur

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1159699
          Files :

          • /hadoop/common/trunk/hadoop-common/src/test/java/org/apache/hadoop/io/AvroTestUtil.java
          • /hadoop/common/trunk/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestAvroRpc.java
          • /hadoop/common/trunk/hadoop-common/src/test/java/org/apache/hadoop/ipc/AvroTestProtocol.java
          • /hadoop/common/trunk/hadoop-common/src/main/java/org/apache/hadoop/ipc/AvroRpcEngine.java
          • /hadoop/common/trunk/hadoop-common/src/test/avro
          • /hadoop/common/trunk/hadoop-common/src/test/avro/avroRecord.avsc
          • /hadoop/common/trunk/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common/src/test/java/org/apache/hadoop/ipc/AvroSpecificTestProtocol.avpr
          • /hadoop/common/trunk/hadoop-common/src/main/java/org/apache/hadoop/io/serializer/avro/AvroSerialization.java
          • /hadoop/common/trunk/hadoop-common/src/test/java/org/apache/hadoop/io/serializer/avro/avroRecord.avsc
          • /hadoop/common/trunk/hadoop-common/src/main/java/org/apache/hadoop/ipc/AvroSpecificRpcEngine.java
          • /hadoop/common/trunk/hadoop-common/pom.xml
          • /hadoop/common/trunk/hadoop-common/src/test/avro/AvroSpecificTestProtocol.avpr
          • /hadoop/common/trunk/hadoop-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #757 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/757/ ) HADOOP-7264 . Bump avro version to at least 1.4.1. Contributed by Alejandro Abdelnur tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1159699 Files : /hadoop/common/trunk/hadoop-common/src/test/java/org/apache/hadoop/io/AvroTestUtil.java /hadoop/common/trunk/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestAvroRpc.java /hadoop/common/trunk/hadoop-common/src/test/java/org/apache/hadoop/ipc/AvroTestProtocol.java /hadoop/common/trunk/hadoop-common/src/main/java/org/apache/hadoop/ipc/AvroRpcEngine.java /hadoop/common/trunk/hadoop-common/src/test/avro /hadoop/common/trunk/hadoop-common/src/test/avro/avroRecord.avsc /hadoop/common/trunk/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common/src/test/java/org/apache/hadoop/ipc/AvroSpecificTestProtocol.avpr /hadoop/common/trunk/hadoop-common/src/main/java/org/apache/hadoop/io/serializer/avro/AvroSerialization.java /hadoop/common/trunk/hadoop-common/src/test/java/org/apache/hadoop/io/serializer/avro/avroRecord.avsc /hadoop/common/trunk/hadoop-common/src/main/java/org/apache/hadoop/ipc/AvroSpecificRpcEngine.java /hadoop/common/trunk/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common/src/test/avro/AvroSpecificTestProtocol.avpr /hadoop/common/trunk/hadoop-project/pom.xml
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12490970/HADOOP-7264v6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 21 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/58//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/58//artifact/trunk/target/newPatchFindbugsWarningshadoop-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/58//artifact/trunk/target/newPatchFindbugsWarningshadoop-annotations.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/58//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12490970/HADOOP-7264v6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 21 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/58//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/58//artifact/trunk/target/newPatchFindbugsWarningshadoop-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/58//artifact/trunk/target/newPatchFindbugsWarningshadoop-annotations.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/58//console This message is automatically generated.
          Hide
          Tom White added a comment -

          +1

          Show
          Tom White added a comment - +1
          Hide
          Alejandro Abdelnur added a comment -

          v5 was incorrectly moving an avro file from MR, v6 fixes that

          Show
          Alejandro Abdelnur added a comment - v5 was incorrectly moving an avro file from MR, v6 fixes that
          Hide
          Tom White added a comment -

          I notice the latest patch moves MapReduce Avro protocol files into Common. Shouldn't they stay in the MapReduce module?

          Show
          Tom White added a comment - I notice the latest patch moves MapReduce Avro protocol files into Common. Shouldn't they stay in the MapReduce module?
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12490927/HADOOP-7264v5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 23 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/55//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/55//artifact/trunk/target/newPatchFindbugsWarningshadoop-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/55//artifact/trunk/target/newPatchFindbugsWarningshadoop-annotations.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/55//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12490927/HADOOP-7264v5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 23 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/55//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/55//artifact/trunk/target/newPatchFindbugsWarningshadoop-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/55//artifact/trunk/target/newPatchFindbugsWarningshadoop-annotations.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/55//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          'v5' follows Luke & Scott recommendation.

          Moved the *.avpr and *.avsc files to src/test/avro/.

          The plugin config is simpler.

          The only thing I've left is that the generate d java code goes to target/generated-test-sources/java (instead leaving the default target/generated-test-sources/avro directory).

          The reason for this is that in Eclipse, the generated*sources/<DIR> directories have to be manually added to the IDE directory and by keeping java it is a single dir that has to be added instead multiple.

          Show
          Alejandro Abdelnur added a comment - 'v5' follows Luke & Scott recommendation. Moved the *.avpr and *.avsc files to src/test/avro/. The plugin config is simpler. The only thing I've left is that the generate d java code goes to target/generated-test-sources/java (instead leaving the default target/generated-test-sources/avro directory). The reason for this is that in Eclipse, the generated*sources/<DIR> directories have to be manually added to the IDE directory and by keeping java it is a single dir that has to be added instead multiple.
          Hide
          Alejandro Abdelnur added a comment -

          Scott, I like that better. Still, we want to do this as part of this patch or as a follow up that puts things in better places not only for avro but for protobuffs, etc, etc?

          Show
          Alejandro Abdelnur added a comment - Scott, I like that better. Still, we want to do this as part of this patch or as a follow up that puts things in better places not only for avro but for protobuffs, etc, etc?
          Hide
          Scott Carey added a comment -

          Maven convention is to have each type of source in its own tree. avro-maven-plugin will work if you point it elsewhere, but some other tools might not.

          Show
          Scott Carey added a comment - Maven convention is to have each type of source in its own tree. avro-maven-plugin will work if you point it elsewhere, but some other tools might not.
          Hide
          Alejandro Abdelnur added a comment -

          I'm OK either way.

          Actually I prefer your approach (I didn't see your first patch until I've uploaded my first version).

          My motivation for doing the way it is in my patches was that I'm trying to be as little invasive as possible for all Mavenization changes, seen many things that could be done better by rearranging things. I just didn't want to add more noise than the strictly necessary during the Mavenization.

          But, if strong about doing this for this JIRA, I can update the patch.

          Show
          Alejandro Abdelnur added a comment - I'm OK either way. Actually I prefer your approach (I didn't see your first patch until I've uploaded my first version). My motivation for doing the way it is in my patches was that I'm trying to be as little invasive as possible for all Mavenization changes, seen many things that could be done better by rearranging things. I just didn't want to add more noise than the strictly necessary during the Mavenization. But, if strong about doing this for this JIRA, I can update the patch.
          Hide
          Luke Lu added a comment -

          I just noticed that v4 (since v2) keeps avro source (*.av

          {sc,pr}

          ) in java source directory, which requires significant explicit configuration of avro-maven-plugin, unlike in MRv2, all avro source is in avro's source directory, which I (and Sidd) think, is more convenient and conforming to maven conventions, which means default configuration would work with the plugin. But I can see some benefit of keep foreign source in java source as well. What other people think about this?

          Show
          Luke Lu added a comment - I just noticed that v4 (since v2) keeps avro source (*.av {sc,pr} ) in java source directory, which requires significant explicit configuration of avro-maven-plugin, unlike in MRv2, all avro source is in avro's source directory, which I (and Sidd) think, is more convenient and conforming to maven conventions, which means default configuration would work with the plugin. But I can see some benefit of keep foreign source in java source as well. What other people think about this?
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12490887/HADOOP-7264v4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 20 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/50//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/50//artifact/trunk/target/newPatchFindbugsWarningshadoop-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/50//artifact/trunk/target/newPatchFindbugsWarningshadoop-annotations.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/50//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12490887/HADOOP-7264v4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 20 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/50//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/50//artifact/trunk/target/newPatchFindbugsWarningshadoop-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/50//artifact/trunk/target/newPatchFindbugsWarningshadoop-annotations.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/50//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          'v4' fixing failing tests

          Show
          Alejandro Abdelnur added a comment - 'v4' fixing failing tests
          Hide
          Tom White added a comment -

          Sorry, I missed those failures. It looks like the problem is that AvroTestUtil uses a buffered binary encoder which isn't properly flushed. It would be simpler to use a direct binary encoder in this case.

          Show
          Tom White added a comment - Sorry, I missed those failures. It looks like the problem is that AvroTestUtil uses a buffered binary encoder which isn't properly flushed. It would be simpler to use a direct binary encoder in this case.
          Hide
          Luke Lu added a comment -

          There're 3 avro related test failures

          Show
          Luke Lu added a comment - There're 3 avro related test failures
          Hide
          Tom White added a comment -

          +1 This looks good to me. I'm assuming that the MapReduce portion will be handled by MAPREDUCE-2848.

          Show
          Tom White added a comment - +1 This looks good to me. I'm assuming that the MapReduce portion will be handled by MAPREDUCE-2848 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12490821/HADOOP-7264v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 20 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/48//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/48//artifact/trunk/target/newPatchFindbugsWarningshadoop-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/48//artifact/trunk/target/newPatchFindbugsWarningshadoop-annotations.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/48//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12490821/HADOOP-7264v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 20 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/48//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/48//artifact/trunk/target/newPatchFindbugsWarningshadoop-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/48//artifact/trunk/target/newPatchFindbugsWarningshadoop-annotations.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/48//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          'v3' version, rebased to current trunk (post MR279), leaving outside upgrade of of other components (slf4j)

          Show
          Alejandro Abdelnur added a comment - 'v3' version, rebased to current trunk (post MR279), leaving outside upgrade of of other components (slf4j)
          Hide
          Alejandro Abdelnur added a comment -

          @Scott, if avro classes are used directly in hadoop-common then an explicit dependency is the correct way of doing it.

          Show
          Alejandro Abdelnur added a comment - @Scott, if avro classes are used directly in hadoop-common then an explicit dependency is the correct way of doing it.
          Hide
          Luke Lu added a comment -

          Looks like the patch is probably out of date due to HADOOP-7536 and INFRA-3857.

          @Scott, yes, avro-ipc should suffice.

          Show
          Luke Lu added a comment - Looks like the patch is probably out of date due to HADOOP-7536 and INFRA-3857 . @Scott, yes, avro-ipc should suffice.
          Hide
          Scott Carey added a comment -

          Avro-1.5.x also allows you to pick and choose specific dependency subsets as needed, while all were in one jar file before.

          avro-ipc seems to be used. If so the dependency on avro is not needed (avro is an implicit transitive dependency of avro-ipc).

          Show
          Scott Carey added a comment - Avro-1.5.x also allows you to pick and choose specific dependency subsets as needed, while all were in one jar file before. avro-ipc seems to be used. If so the dependency on avro is not needed (avro is an implicit transitive dependency of avro-ipc).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12490572/HADOOP-7264v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 20 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/41//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12490572/HADOOP-7264v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 20 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/41//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          Updated patch for common using Avro 1.5.2.

          Remove unnecessary/unrelated changes to the POMs

          Show
          Alejandro Abdelnur added a comment - Updated patch for common using Avro 1.5.2. Remove unnecessary/unrelated changes to the POMs
          Hide
          Tom White added a comment -

          Avro 1.5.2 has just been released today, so it would be good to use that version.

          Show
          Tom White added a comment - Avro 1.5.2 has just been released today, so it would be good to use that version.
          Hide
          Luke Lu added a comment -

          TestLeafQueue doesn't pass now in MR-279, which seems to be unrelated. Arun is looking at it.

          Show
          Luke Lu added a comment - TestLeafQueue doesn't pass now in MR-279, which seems to be unrelated. Arun is looking at it.
          Hide
          Luke Lu added a comment -

          v1 patch took the bump avro to the current 1.5.1. Since it'd be an incompatible change to bump to 1.4.1 anyway, we might as well upgrade the current stable version with an official maven plugin.

          Show
          Luke Lu added a comment - v1 patch took the bump avro to the current 1.5.1. Since it'd be an incompatible change to bump to 1.4.1 anyway, we might as well upgrade the current stable version with an official maven plugin.
          Hide
          Luke Lu added a comment -

          mr2 v1 patch is the for MR-279 branch only.

          Show
          Luke Lu added a comment - mr2 v1 patch is the for MR-279 branch only.
          Hide
          Arun C Murthy added a comment -

          Luke, we do have the patch for 'classic' MR trunk in y-merge, can you pls port it and commit? Thanks.

          This will ensure nothing breaks... with or without the MR-279 merge.

          Show
          Arun C Murthy added a comment - Luke, we do have the patch for 'classic' MR trunk in y-merge, can you pls port it and commit? Thanks. This will ensure nothing breaks... with or without the MR-279 merge.
          Hide
          Luke Lu added a comment -

          BTW, HADOOP-6671 bumped versions of many packages (jetty, jackson, slf4j) with no ill-effect yet . The avro bump from 1.3+ to 1.4+ will break the build of trunk mapreduce jobhistory, which will be replaced by MR-279, which is already on 1.4+.

          Show
          Luke Lu added a comment - BTW, HADOOP-6671 bumped versions of many packages (jetty, jackson, slf4j) with no ill-effect yet . The avro bump from 1.3+ to 1.4+ will break the build of trunk mapreduce jobhistory, which will be replaced by MR-279, which is already on 1.4+.
          Hide
          Todd Lipcon added a comment -

          Great news

          Show
          Todd Lipcon added a comment - Great news
          Hide
          Arun C Murthy added a comment -

          Todd in MR2, the NM's deps don't leak to the containers (tasks).

          Show
          Arun C Murthy added a comment - Todd in MR2, the NM's deps don't leak to the containers (tasks).
          Hide
          Arun C Murthy added a comment -

          +1 for bumping - MR moved to 1.4.* long ago.

          Show
          Arun C Murthy added a comment - +1 for bumping - MR moved to 1.4.* long ago.
          Hide
          Luke Lu added a comment -

          We do want to improve the isolation of dependencies between framework and user code, which I believe is related but really orthogonal to this jira. HADOOP-7150 is trying to achieve that. I believe it's mostly achievable via careful refactor of classpath construction and a little shade/jar/jar in the near term and completely nailed with some like OSGi in the long term.

          Show
          Luke Lu added a comment - We do want to improve the isolation of dependencies between framework and user code, which I believe is related but really orthogonal to this jira. HADOOP-7150 is trying to achieve that. I believe it's mostly achievable via careful refactor of classpath construction and a little shade/jar/jar in the near term and completely nailed with some like OSGi in the long term.
          Hide
          Todd Lipcon added a comment -

          k, thanks for the perspective. I bring this up because we've heard from a number of users that, when we change the version of dependencies like thrift, avro, or (especially) Jackson, it hurts users who use these same libraries in their jobs.

          Show
          Todd Lipcon added a comment - k, thanks for the perspective. I bring this up because we've heard from a number of users that, when we change the version of dependencies like thrift, avro, or (especially) Jackson, it hurts users who use these same libraries in their jobs.
          Hide
          Luke Lu added a comment -

          maven shade and jarjar doesn't help in this case for mr2 (it's in the common code in job history), as both client and server needs to be on at least 1.4.1, sans an OSGi containment in the job history server, which would be over-engineering at this point. maven shade/jarjar would be helpful for future client libraries.

          BTW, we initially actually used maven shade for various servers, but had to remove it as the added processing time, the annoying warnings and the resulting bulk really don't worth the trouble.

          Show
          Luke Lu added a comment - maven shade and jarjar doesn't help in this case for mr2 (it's in the common code in job history), as both client and server needs to be on at least 1.4.1, sans an OSGi containment in the job history server, which would be over-engineering at this point. maven shade/jarjar would be helpful for future client libraries. BTW, we initially actually used maven shade for various servers, but had to remove it as the added processing time, the annoying warnings and the resulting bulk really don't worth the trouble.
          Hide
          Todd Lipcon added a comment -

          Is it worth considering maven shade or jarjar for avro? Or in MR2 do we no longer have the issue that the daemon formerly known as the TaskTracker leaks its jar dependencies onto the user classpath?

          Show
          Todd Lipcon added a comment - Is it worth considering maven shade or jarjar for avro? Or in MR2 do we no longer have the issue that the daemon formerly known as the TaskTracker leaks its jar dependencies onto the user classpath?

            People

            • Assignee:
              Luke Lu
              Reporter:
              Luke Lu
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development