Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1917

Clean up duplication of dependent jar files

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: build
    • Labels:
      None
    • Environment:

      Java 6, RHEL 5.5

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Remove packaging of duplicated third party jar files

      Description

      For trunk, the build and deployment tree look like this:

      hadoop-common-0.2x.y
      hadoop-hdfs-0.2x.y
      hadoop-mapred-0.2x.y

      Technically, hdfs's the third party dependent jar files should be fetch from hadoop-common. However, it is currently fetching from hadoop-hdfs/lib only. It would be nice to eliminate the need to repeat duplicated jar files at build time.

      There are two options to manage this dependency list, continue to enhance ant build structure to fetch and filter jar file dependencies using ivy. On the other hand, it would be a good opportunity to convert the build structure to maven, and use maven to manage the provided jar files.

      1. HDFS-1917-1.patch
        6 kB
        Eric Yang
      2. HDFS-1917.patch
        6 kB
        Eric Yang

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #658 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/658/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #658 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/658/ )
          Hide
          Eric Yang added a comment -

          ivy-retrieve-common should not depends on ivy-retrieve-hdfs. It becomes a circular dependency. It looks like ivy-retrieve-hdfs is not initialized when calling test-system. I will track it down, thanks Todd.

          Show
          Eric Yang added a comment - ivy-retrieve-common should not depends on ivy-retrieve-hdfs. It becomes a circular dependency. It looks like ivy-retrieve-hdfs is not initialized when calling test-system. I will track it down, thanks Todd.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Todd, good catch. Filed HDFS-1938 and Eric is looking at it.

          Show
          Tsz Wo Nicholas Sze added a comment - Todd, good catch. Filed HDFS-1938 and Eric is looking at it.
          Hide
          Todd Lipcon added a comment -

          Jolly points out that making ivy-retrieve-common depend on ivy-retrieve-hdfs fixes the issue. Does that seem right to you guys?

          Show
          Todd Lipcon added a comment - Jolly points out that making ivy-retrieve-common depend on ivy-retrieve-hdfs fixes the issue. Does that seem right to you guys?
          Hide
          Todd Lipcon added a comment -

          Not sure why the testbot didn't catch this, but it looks like this commit broke the system test compilation. Seeing "Reference ivy-hdfs.classpath not found." when I run ant test-system. Reverting this patch locally fixes it.

          Show
          Todd Lipcon added a comment - Not sure why the testbot didn't catch this, but it looks like this commit broke the system test compilation. Seeing "Reference ivy-hdfs.classpath not found." when I run ant test-system. Reverting this patch locally fixes it.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Eric!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Eric!
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Eric Yang added a comment -

          Quote from HDFS-1903:

          These test changes correspond to changes in HADOOP-7267. The TestHDFSCLI and TestDFSShell errors occurred because the build started before the hadoop bug was committed. The flux in these tests should wind down in a few days.

          The test fail is not related to this patch.

          Show
          Eric Yang added a comment - Quote from HDFS-1903 : These test changes correspond to changes in HADOOP-7267 . The TestHDFSCLI and TestDFSShell errors occurred because the build started before the hadoop bug was committed. The flux in these tests should wind down in a few days. The test fail is not related to this patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479013/HDFS-1917-1.patch
          against trunk revision 1102467.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestHDFSCLI
          org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
          org.apache.hadoop.hdfs.TestFileConcurrentReader
          org.apache.hadoop.tools.TestJMXGet

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/507//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/507//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/507//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479013/HDFS-1917-1.patch against trunk revision 1102467. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.tools.TestJMXGet +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/507//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/507//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/507//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479007/HDFS-1917.patch
          against trunk revision 1102153.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestHDFSCLI
          org.apache.hadoop.hdfs.server.namenode.TestBlocksWithNotEnoughRacks
          org.apache.hadoop.hdfs.TestDFSShell
          org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
          org.apache.hadoop.hdfs.TestFileConcurrentReader
          org.apache.hadoop.tools.TestJMXGet

          +1 contrib tests. The patch passed contrib unit tests.

          -1 system test framework. The patch failed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/506//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/506//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/506//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479007/HDFS-1917.patch against trunk revision 1102153. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.namenode.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.tools.TestJMXGet +1 contrib tests. The patch passed contrib unit tests. -1 system test framework. The patch failed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/506//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/506//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/506//console This message is automatically generated.
          Hide
          Eric Yang added a comment -

          Revise patch to add hdfs ivy configuration. Thanks Luke!

          Show
          Eric Yang added a comment - Revise patch to add hdfs ivy configuration. Thanks Luke!
          Hide
          Luke Lu added a comment -

          Though I understand the goal is to separate the hdfs only dependencies for easier dedup, it seems to me that if you keep the common profile as is and add an hdfs profile for common-daemon. The patch would be smaller and less confusing (the common profile now contains hdfs only dependencies and the compile profile is actually from common.)

          Show
          Luke Lu added a comment - Though I understand the goal is to separate the hdfs only dependencies for easier dedup, it seems to me that if you keep the common profile as is and add an hdfs profile for common-daemon. The patch would be smaller and less confusing (the common profile now contains hdfs only dependencies and the compile profile is actually from common.)
          Hide
          Eric Yang added a comment -

          Remove packaging of duplicated third party jar files

          Show
          Eric Yang added a comment - Remove packaging of duplicated third party jar files
          Hide
          Eric Yang added a comment -
          • Changed ivy configuration to setup third party jar file for compile profile.
          • common profile contains only common-daemon to be included in HADOOP_HOME/lib.
          Show
          Eric Yang added a comment - Changed ivy configuration to setup third party jar file for compile profile. common profile contains only common-daemon to be included in HADOOP_HOME/lib.
          Hide
          Eric Yang added a comment -

          Patch for this jira is going to assume that hadoop-common third party jar files can be referenced from HADOOP_HOME/lib until HADOOP-6255 and proposed HADOOP_PREFIX take place. Where HADOOP_HOME is the PREFIX directory of hadoop-common-0.2x.y = hadoop-hdfs-0.2x.y = hadoop-mapred-0.2x.y.

          Show
          Eric Yang added a comment - Patch for this jira is going to assume that hadoop-common third party jar files can be referenced from HADOOP_HOME/lib until HADOOP-6255 and proposed HADOOP_PREFIX take place. Where HADOOP_HOME is the PREFIX directory of hadoop-common-0.2x.y = hadoop-hdfs-0.2x.y = hadoop-mapred-0.2x.y.

            People

            • Assignee:
              Eric Yang
              Reporter:
              Eric Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development