Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Removed duplicated hadoop-common library dependencies.

      Description

      For trunk, the build and deployment tree look like this:

      hadoop-common-0.2x.y
      hadoop-hdfs-0.2x.y
      hadoop-mapred-0.2x.y

      Technically, mapred's the third party dependent jar files should be fetch from hadoop-common and hadoop-hdfs. However, it is currently fetching from hadoop-mapred/lib only. It would be nice to eliminate the need to repeat duplicated jar files at build time.

      There are two options to manage this dependency list, continue to enhance ant build structure to fetch and filter jar file dependencies using ivy. On the other hand, it would be a good opportunity to convert the build structure to maven, and use maven to manage the provided jar files.

        Issue Links

          Activity

          Eric Yang created issue -
          Eric Yang made changes -
          Field Original Value New Value
          Link This issue is blocked by HADOOP-6255 [ HADOOP-6255 ]
          Eric Yang made changes -
          Link This issue is blocked by HADOOP-6255 [ HADOOP-6255 ]
          Eric Yang made changes -
          Link This issue blocks HADOOP-6255 [ HADOOP-6255 ]
          Hide
          Eric Yang added a comment -

          For trunk, the third party jar files are duplicated 3 times across common, hdfs, mapred. We should clean up the dependency and build structure to reduce duplications for deployment.

          Show
          Eric Yang added a comment - For trunk, the third party jar files are duplicated 3 times across common, hdfs, mapred. We should clean up the dependency and build structure to reduce duplications for deployment.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          There are also duplicated dependent jars from codes and tests (e.g the same jar will appear in both build/ivy/lib/hadoop-hdfs/common/ and build/ivy/lib/hadoop-hdfs/test/). See if you want to fix it here.

          Show
          Tsz Wo Nicholas Sze added a comment - There are also duplicated dependent jars from codes and tests (e.g the same jar will appear in both build/ivy/lib/hadoop-hdfs/common/ and build/ivy/lib/hadoop-hdfs/test/ ). See if you want to fix it here.
          Hide
          Eric Yang added a comment -

          Nicholas, I have a patch for hdfs in HDFS-1917. It would be really helpful, if you can review it. Thanks

          Show
          Eric Yang added a comment - Nicholas, I have a patch for hdfs in HDFS-1917 . It would be really helpful, if you can review it. Thanks
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Eric, done.

          Show
          Tsz Wo Nicholas Sze added a comment - Eric, done.
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HADOOP-7289 [ HADOOP-7289 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Eric, with you idea, I filed HADOOP-7289. Could you review it?

          Show
          Tsz Wo Nicholas Sze added a comment - Eric, with you idea, I filed HADOOP-7289 . Could you review it?
          Hide
          Eric Yang added a comment -

          Nicholas, done.

          Show
          Eric Yang added a comment - Nicholas, done.
          Hide
          Eric Yang added a comment -
          • Removed duplicated dependency for hadoop common libraries.
          • Fix a bug in mumak overwriting mapred bundled library.
          Show
          Eric Yang added a comment - Removed duplicated dependency for hadoop common libraries. Fix a bug in mumak overwriting mapred bundled library.
          Eric Yang made changes -
          Attachment MAPREDUCE-2483.patch [ 12479170 ]
          Eric Yang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note Removed duplicated hadoop-common library dependencies.
          Assignee Eric Yang [ eyang ]
          Eric Yang made changes -
          Link This issue incorporates MAPREDUCE-2496 [ MAPREDUCE-2496 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479170/MAPREDUCE-2483.patch
          against trunk revision 1103921.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/252//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/252//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/252//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479170/MAPREDUCE-2483.patch against trunk revision 1103921. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/252//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/252//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/252//console This message is automatically generated.
          Hide
          Owen O'Malley added a comment -

          I just committed this. Thanks, Eric!

          Show
          Owen O'Malley added a comment - I just committed this. Thanks, Eric!
          Owen O'Malley made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.23.0 [ 12315570 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #685 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/685/)
          MAPREDUCE-2483. Remove duplication of jars between Hadoop subprojects
          from build artifacts. (Eric Yang via omalley)

          omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125017
          Files :

          • /hadoop/mapreduce/trunk/CHANGES.txt
          • /hadoop/mapreduce/trunk/ivy.xml
          • /hadoop/mapreduce/trunk/src/contrib/mumak/build.xml
          • /hadoop/mapreduce/trunk/build.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #685 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/685/ ) MAPREDUCE-2483 . Remove duplication of jars between Hadoop subprojects from build artifacts. (Eric Yang via omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125017 Files : /hadoop/mapreduce/trunk/CHANGES.txt /hadoop/mapreduce/trunk/ivy.xml /hadoop/mapreduce/trunk/src/contrib/mumak/build.xml /hadoop/mapreduce/trunk/build.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #685 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/685/)
          MAPREDUCE-2483. Remove duplication of jars between Hadoop subprojects
          from build artifacts. (Eric Yang via omalley)

          omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125017
          Files :

          • /hadoop/mapreduce/trunk/CHANGES.txt
          • /hadoop/mapreduce/trunk/ivy.xml
          • /hadoop/mapreduce/trunk/src/contrib/mumak/build.xml
          • /hadoop/mapreduce/trunk/build.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #685 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/685/ ) MAPREDUCE-2483 . Remove duplication of jars between Hadoop subprojects from build artifacts. (Eric Yang via omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125017 Files : /hadoop/mapreduce/trunk/CHANGES.txt /hadoop/mapreduce/trunk/ivy.xml /hadoop/mapreduce/trunk/src/contrib/mumak/build.xml /hadoop/mapreduce/trunk/build.xml
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Eric Yang
              Reporter:
              Eric Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development