Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2196

Make ant build system work with hadoop-common JAR generated by Maven

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Some tweaks must be done in HDFS ivy configuration to work with HADOOP-6671.

      This wil be a temporary fix until HFDS is mavenized.

      1. a1-HDFS-2196.patch
        3 kB
        Alejandro Abdelnur
      2. HDFS-2196.patch
        2 kB
        Tom White
      3. h2196_20110804.patch
        0.6 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          Alejandro Abdelnur added a comment -

          Based on patch file Tom White added to HADOOP-6671

          Show
          Alejandro Abdelnur added a comment - Based on patch file Tom White added to HADOOP-6671
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12487777/a1-HDFS-2196.patch
          against trunk revision 1150960.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:

          -1 contrib tests. The patch failed contrib unit tests.

          -1 system test framework. The patch failed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1010//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1010//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12487777/a1-HDFS-2196.patch against trunk revision 1150960. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: -1 contrib tests. The patch failed contrib unit tests. -1 system test framework. The patch failed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1010//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1010//console This message is automatically generated.
          Hide
          Tom White added a comment -

          This new patch removes the HDFS proxy changes in the previous patch since that component has been removed now.

          I verified that with HADOOP-6671 and this patch I could build HDFS ('ant veryclean jar -Dresolvers=internal') using local Common jars built with 'mvn install'.

          Note that this cannot be committed until after HADOOP-6671 since it introduces a dependency on hadoop-annotations introduced in that JIRA. This is also what caused the QA bot to fail.

          Show
          Tom White added a comment - This new patch removes the HDFS proxy changes in the previous patch since that component has been removed now. I verified that with HADOOP-6671 and this patch I could build HDFS ('ant veryclean jar -Dresolvers=internal') using local Common jars built with 'mvn install'. Note that this cannot be committed until after HADOOP-6671 since it introduces a dependency on hadoop-annotations introduced in that JIRA. This is also what caused the QA bot to fail.
          Hide
          Tom White added a comment -

          I've committed this. Thanks, Alejandro.

          Show
          Tom White added a comment - I've committed this. Thanks, Alejandro.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Failed to run "ant test-patch"

          [ivy:resolve] :: problems summary ::[ivy:resolve] :::: WARNINGS
          [ivy:resolve]           module not found: org.apache.hadoop#hadoop-annotations;0.23.0-SNAPSHOT[ivy:resolve]   ==== apache-snapshot: tried
          [ivy:resolve]     https://repository.apache.org/content/repositories/snapshots/o
          rg/apache/hadoop/hadoop-annotations/0.23.0-SNAPSHOT/hadoop-annotations-0.23.0-SNAPSHOT.pom
          [ivy:resolve]     -- artifact org.apache.hadoop#hadoop-annotations;0.23.0-SNAPSHOT!hadoop-annotations.jar:
          [ivy:resolve]     https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-annotations/0.23.0-SNAPSHOT/hadoop-annotations-0.23.0-SN
          APSHOT.jar
          [ivy:resolve]   ==== maven2: tried
          [ivy:resolve]     http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-annotat
          ions/0.23.0-SNAPSHOT/hadoop-annotations-0.23.0-SNAPSHOT.pom[ivy:resolve]     -- artifact org.apache.hadoop#hadoop-annotations;0.23.0-SNAPSH
          OT!hadoop-annotations.jar:
          [ivy:resolve]     http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-annotat
          ions/0.23.0-SNAPSHOT/hadoop-annotations-0.23.0-SNAPSHOT.jar
          [ivy:resolve]           ::::::::::::::::::::::::::::::::::::::::::::::
          [ivy:resolve]           ::          UNRESOLVED DEPENDENCIES         ::
          [ivy:resolve]           ::::::::::::::::::::::::::::::::::::::::::::::[ivy:resolve]           :: org.apache.hadoop#hadoop-annotations;0.23.0-SNAPSHOT:
           not found
          [ivy:resolve]           ::::::::::::::::::::::::::::::::::::::::::::::
          [ivy:resolve]
          [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
          
          BUILD FAILED
          /Users/szetszwo/hadoop/test-patch/hdfs/build.xml:1966: impossible to resolve dep
          endencies:
                  resolve failed - see output for details
          
          Total time: 24 seconds
          
          Show
          Tsz Wo Nicholas Sze added a comment - Failed to run "ant test-patch" [ivy:resolve] :: problems summary ::[ivy:resolve] :::: WARNINGS [ivy:resolve] module not found: org.apache.hadoop#hadoop-annotations;0.23.0-SNAPSHOT[ivy:resolve] ==== apache-snapshot: tried [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/o rg/apache/hadoop/hadoop-annotations/0.23.0-SNAPSHOT/hadoop-annotations-0.23.0-SNAPSHOT.pom [ivy:resolve] -- artifact org.apache.hadoop#hadoop-annotations;0.23.0-SNAPSHOT!hadoop-annotations.jar: [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-annotations/0.23.0-SNAPSHOT/hadoop-annotations-0.23.0-SN APSHOT.jar [ivy:resolve] ==== maven2: tried [ivy:resolve] http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-annotat ions/0.23.0-SNAPSHOT/hadoop-annotations-0.23.0-SNAPSHOT.pom[ivy:resolve] -- artifact org.apache.hadoop#hadoop-annotations;0.23.0-SNAPSH OT!hadoop-annotations.jar: [ivy:resolve] http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-annotat ions/0.23.0-SNAPSHOT/hadoop-annotations-0.23.0-SNAPSHOT.jar [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] :: UNRESOLVED DEPENDENCIES :: [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::[ivy:resolve] :: org.apache.hadoop#hadoop-annotations;0.23.0-SNAPSHOT: not found [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS BUILD FAILED /Users/szetszwo/hadoop/test-patch/hdfs/build.xml:1966: impossible to resolve dep endencies: resolve failed - see output for details Total time: 24 seconds
          Hide
          Tsz Wo Nicholas Sze added a comment -

          "ant" also failed.

          Show
          Tsz Wo Nicholas Sze added a comment - "ant" also failed.
          Hide
          Alejandro Abdelnur added a comment -

          Nicholas,

          As Tom commented in HADOOP-6671, you need to run 'mvn install -DskipTests' from trunk first, then HDFS should work with Maven.

          Note that this is a transient requirement until the artifacts from Hadoop common are published to the Apache Snapshots repo (opening a JIRA for that now).

          Show
          Alejandro Abdelnur added a comment - Nicholas, As Tom commented in HADOOP-6671 , you need to run 'mvn install -DskipTests' from trunk first, then HDFS should work with Maven. Note that this is a transient requirement until the artifacts from Hadoop common are published to the Apache Snapshots repo (opening a JIRA for that now).
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Alejandro,

          The mvn command also did not work; see this. I understand that this is temporary. When do you expect to fix it?

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Alejandro, The mvn command also did not work; see this . I understand that this is temporary. When do you expect to fix it?
          Hide
          Tom White added a comment -

          I have just deployed the artifacts to the Apache Snapshot repo, so HDFS and MapReduce both build without doing a Maven build first. (See HADOOP-7501.)

          Show
          Tom White added a comment - I have just deployed the artifacts to the Apache Snapshot repo, so HDFS and MapReduce both build without doing a Maven build first. (See HADOOP-7501 .)
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Just tried. HDFS can be compiled now. Thanks.

          Show
          Tsz Wo Nicholas Sze added a comment - Just tried. HDFS can be compiled now. Thanks.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > ... HDFS can be compiled now. ...

          However, fuse-dfs cannot be compiled.

          [ivy:resolve] 	  http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
          [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
          [ivy:resolve] 		::              FAILED DOWNLOADS            ::
          [ivy:resolve] 		:: ^ see resolution messages for details  ^ ::
          [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
          [ivy:resolve] 		:: javax.jms#jms;1.1!jms.jar
          [ivy:resolve] 		:: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
          [ivy:resolve] 		:: com.sun.jmx#jmxri;1.2.1!jmxri.jar
          [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
          [ivy:resolve] 
          [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
          
          Show
          Tsz Wo Nicholas Sze added a comment - > ... HDFS can be compiled now. ... However, fuse-dfs cannot be compiled. [ivy:resolve] http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] :: FAILED DOWNLOADS :: [ivy:resolve] :: ^ see resolution messages for details ^ :: [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] :: javax.jms#jms;1.1!jms.jar [ivy:resolve] :: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar [ivy:resolve] :: com.sun.jmx#jmxri;1.2.1!jmxri.jar [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h2196_20110804.patch: fix ivy.xml in fuse-dfs.

          Show
          Tsz Wo Nicholas Sze added a comment - h2196_20110804.patch: fix ivy.xml in fuse-dfs.
          Hide
          Tom White added a comment -

          +1 I committed the fuse-dfs fix. Thanks, Nicholas.

          Show
          Tom White added a comment - +1 I committed the fuse-dfs fix. Thanks, Nicholas.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #738 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/738/)
          HDFS-2196. Make ant build system work with hadoop-common JAR generated by Maven. Fix for fusedfs contributed by szetszwo
          HDFS-2196. Make ant build system work with hadoop-common JAR generated by Maven. Contributed by Alejandro Abdelnur

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153928
          Files :

          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/ivy.xml

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153207
          Files :

          • /hadoop/common/trunk/hdfs/CHANGES.txt
          • /hadoop/common/trunk/hdfs/ivy.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #738 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/738/ ) HDFS-2196 . Make ant build system work with hadoop-common JAR generated by Maven. Fix for fusedfs contributed by szetszwo HDFS-2196 . Make ant build system work with hadoop-common JAR generated by Maven. Contributed by Alejandro Abdelnur tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153928 Files : /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/ivy.xml tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153207 Files : /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/ivy.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #812 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/812/)
          HDFS-2196. Make ant build system work with hadoop-common JAR generated by Maven. Fix for fusedfs contributed by szetszwo
          HDFS-2196. Make ant build system work with hadoop-common JAR generated by Maven. Contributed by Alejandro Abdelnur

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153928
          Files :

          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/ivy.xml

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153207
          Files :

          • /hadoop/common/trunk/hdfs/CHANGES.txt
          • /hadoop/common/trunk/hdfs/ivy.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #812 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/812/ ) HDFS-2196 . Make ant build system work with hadoop-common JAR generated by Maven. Fix for fusedfs contributed by szetszwo HDFS-2196 . Make ant build system work with hadoop-common JAR generated by Maven. Contributed by Alejandro Abdelnur tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153928 Files : /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/ivy.xml tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153207 Files : /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/ivy.xml

            People

            • Assignee:
              Alejandro Abdelnur
              Reporter:
              Alejandro Abdelnur
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development