Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Removed duplicated hadoop-common library dependencies.

      Description

      For trunk, the build and deployment tree look like this:

      hadoop-common-0.2x.y
      hadoop-hdfs-0.2x.y
      hadoop-mapred-0.2x.y

      Technically, mapred's the third party dependent jar files should be fetch from hadoop-common and hadoop-hdfs. However, it is currently fetching from hadoop-mapred/lib only. It would be nice to eliminate the need to repeat duplicated jar files at build time.

      There are two options to manage this dependency list, continue to enhance ant build structure to fetch and filter jar file dependencies using ivy. On the other hand, it would be a good opportunity to convert the build structure to maven, and use maven to manage the provided jar files.

        Issue Links

          Activity

          Hide
          Eric Yang added a comment -

          For trunk, the third party jar files are duplicated 3 times across common, hdfs, mapred. We should clean up the dependency and build structure to reduce duplications for deployment.

          Show
          Eric Yang added a comment - For trunk, the third party jar files are duplicated 3 times across common, hdfs, mapred. We should clean up the dependency and build structure to reduce duplications for deployment.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          There are also duplicated dependent jars from codes and tests (e.g the same jar will appear in both build/ivy/lib/hadoop-hdfs/common/ and build/ivy/lib/hadoop-hdfs/test/). See if you want to fix it here.

          Show
          Tsz Wo Nicholas Sze added a comment - There are also duplicated dependent jars from codes and tests (e.g the same jar will appear in both build/ivy/lib/hadoop-hdfs/common/ and build/ivy/lib/hadoop-hdfs/test/ ). See if you want to fix it here.
          Hide
          Eric Yang added a comment -

          Nicholas, I have a patch for hdfs in HDFS-1917. It would be really helpful, if you can review it. Thanks

          Show
          Eric Yang added a comment - Nicholas, I have a patch for hdfs in HDFS-1917 . It would be really helpful, if you can review it. Thanks
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Eric, done.

          Show
          Tsz Wo Nicholas Sze added a comment - Eric, done.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Eric, with you idea, I filed HADOOP-7289. Could you review it?

          Show
          Tsz Wo Nicholas Sze added a comment - Eric, with you idea, I filed HADOOP-7289 . Could you review it?
          Hide
          Eric Yang added a comment -

          Nicholas, done.

          Show
          Eric Yang added a comment - Nicholas, done.
          Hide
          Eric Yang added a comment -
          • Removed duplicated dependency for hadoop common libraries.
          • Fix a bug in mumak overwriting mapred bundled library.
          Show
          Eric Yang added a comment - Removed duplicated dependency for hadoop common libraries. Fix a bug in mumak overwriting mapred bundled library.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479170/MAPREDUCE-2483.patch
          against trunk revision 1103921.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/252//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/252//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/252//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479170/MAPREDUCE-2483.patch against trunk revision 1103921. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/252//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/252//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/252//console This message is automatically generated.
          Hide
          Owen O'Malley added a comment -

          I just committed this. Thanks, Eric!

          Show
          Owen O'Malley added a comment - I just committed this. Thanks, Eric!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #685 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/685/)
          MAPREDUCE-2483. Remove duplication of jars between Hadoop subprojects
          from build artifacts. (Eric Yang via omalley)

          omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125017
          Files :

          • /hadoop/mapreduce/trunk/CHANGES.txt
          • /hadoop/mapreduce/trunk/ivy.xml
          • /hadoop/mapreduce/trunk/src/contrib/mumak/build.xml
          • /hadoop/mapreduce/trunk/build.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #685 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/685/ ) MAPREDUCE-2483 . Remove duplication of jars between Hadoop subprojects from build artifacts. (Eric Yang via omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125017 Files : /hadoop/mapreduce/trunk/CHANGES.txt /hadoop/mapreduce/trunk/ivy.xml /hadoop/mapreduce/trunk/src/contrib/mumak/build.xml /hadoop/mapreduce/trunk/build.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #685 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/685/)
          MAPREDUCE-2483. Remove duplication of jars between Hadoop subprojects
          from build artifacts. (Eric Yang via omalley)

          omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125017
          Files :

          • /hadoop/mapreduce/trunk/CHANGES.txt
          • /hadoop/mapreduce/trunk/ivy.xml
          • /hadoop/mapreduce/trunk/src/contrib/mumak/build.xml
          • /hadoop/mapreduce/trunk/build.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #685 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/685/ ) MAPREDUCE-2483 . Remove duplication of jars between Hadoop subprojects from build artifacts. (Eric Yang via omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125017 Files : /hadoop/mapreduce/trunk/CHANGES.txt /hadoop/mapreduce/trunk/ivy.xml /hadoop/mapreduce/trunk/src/contrib/mumak/build.xml /hadoop/mapreduce/trunk/build.xml

            People

            • Assignee:
              Eric Yang
              Reporter:
              Eric Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development