Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13961

Fix compilation failure from missing hadoop-kms test jar

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: build
    • Labels:
      None

      Description

      mvn install fails for me on trunk on a new environment with the following:

      [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve dependencies for project org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: Could not find artifact org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-20161228.102554-925 in apache.snapshots.https (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
      

      This works on an existing dev setup, likely because I have the jar in my m2 cache.

      1. HADOOP-13961.002.patch
        2 kB
        Sangjin Lee
      2. HADOOP-13961.001.patch
        2 kB
        John Zhuge

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11110 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11110/)
          HADOOP-13961. Fix compilation failure from missing hadoop-kms test jar. (wang: rev 5f336512d08c0fb74e0404fcde1288926eeba06d)

          • (edit) hadoop-common-project/hadoop-kms/pom.xml
          • (edit) hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • (edit) hadoop-project/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11110 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11110/ ) HADOOP-13961 . Fix compilation failure from missing hadoop-kms test jar. (wang: rev 5f336512d08c0fb74e0404fcde1288926eeba06d) (edit) hadoop-common-project/hadoop-kms/pom.xml (edit) hadoop-hdfs-project/hadoop-hdfs/pom.xml (edit) hadoop-project/pom.xml
          Hide
          jzhuge John Zhuge added a comment -

          Thanks Andrew Wang, Sangjin Lee, Allen Wittenauer, and Karthik Kambatla for reporting the issue.

          Show
          jzhuge John Zhuge added a comment - Thanks Andrew Wang , Sangjin Lee , Allen Wittenauer , and Karthik Kambatla for reporting the issue.
          Hide
          andrew.wang Andrew Wang added a comment -

          Committed to trunk. Thanks for the fast action here John and Sangjin!

          Show
          andrew.wang Andrew Wang added a comment - Committed to trunk. Thanks for the fast action here John and Sangjin!
          Hide
          andrew.wang Andrew Wang added a comment -

          +1 LGTM, did some local testing too. Will commit shortly.

          Show
          andrew.wang Andrew Wang added a comment - +1 LGTM, did some local testing too. Will commit shortly.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 2m 4s Maven dependency ordering for branch
          +1 mvninstall 13m 30s trunk passed
          +1 compile 9m 49s trunk passed
          +1 mvnsite 2m 12s trunk passed
          +1 mvneclipse 0m 50s trunk passed
          +1 javadoc 1m 17s trunk passed
          0 mvndep 0m 38s Maven dependency ordering for patch
          +1 mvninstall 1m 7s the patch passed
          +1 compile 9m 41s the patch passed
          +1 javac 9m 41s the patch passed
          +1 mvnsite 2m 12s the patch passed
          +1 mvneclipse 0m 58s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          +1 javadoc 1m 21s the patch passed
          +1 unit 0m 13s hadoop-project in the patch passed.
          +1 unit 2m 19s hadoop-kms in the patch passed.
          +1 unit 90m 23s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 35s The patch does not generate ASF License warnings.
          141m 5s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13961
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846611/HADOOP-13961.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 05980a4c656e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e692316
          Default Java 1.8.0_111
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11413/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11413/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 2m 4s Maven dependency ordering for branch +1 mvninstall 13m 30s trunk passed +1 compile 9m 49s trunk passed +1 mvnsite 2m 12s trunk passed +1 mvneclipse 0m 50s trunk passed +1 javadoc 1m 17s trunk passed 0 mvndep 0m 38s Maven dependency ordering for patch +1 mvninstall 1m 7s the patch passed +1 compile 9m 41s the patch passed +1 javac 9m 41s the patch passed +1 mvnsite 2m 12s the patch passed +1 mvneclipse 0m 58s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. +1 javadoc 1m 21s the patch passed +1 unit 0m 13s hadoop-project in the patch passed. +1 unit 2m 19s hadoop-kms in the patch passed. +1 unit 90m 23s hadoop-hdfs in the patch passed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 141m 5s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13961 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846611/HADOOP-13961.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 05980a4c656e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e692316 Default Java 1.8.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11413/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11413/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Hit submit patch and changed assignee to Sangjin, can credit both John and Sangjin in the commit message.

          Show
          andrew.wang Andrew Wang added a comment - Hit submit patch and changed assignee to Sangjin, can credit both John and Sangjin in the commit message.
          Hide
          jzhuge John Zhuge added a comment - - edited

          +1 (non-binding) on Sangjin's patch 002.

          Testing done

          • Existing source tree
            1. rm -fr ~/.m2/repository/
            2. ( cd hadoop-maven-plugins/ && mvn install )
            3. mvn clean
            4. mvn install -DskipTests -Dmaven.javadoc.skip -Pnative
          • New source tree
            1. rm -fr ~/.m2/repository/
            2. mvn install -DskipTests -Dmaven.javadoc.skip -Pnative

          I also tried moving MiniKMS from test to main, the pom files would be cleaner, but several test resource files had to be moved as well. I think patch 002 is fine.

          Show
          jzhuge John Zhuge added a comment - - edited +1 (non-binding) on Sangjin's patch 002. Testing done Existing source tree rm -fr ~/.m2/repository/ ( cd hadoop-maven-plugins/ && mvn install ) mvn clean mvn install -DskipTests -Dmaven.javadoc.skip -Pnative New source tree rm -fr ~/.m2/repository/ mvn install -DskipTests -Dmaven.javadoc.skip -Pnative I also tried moving MiniKMS from test to main, the pom files would be cleaner, but several test resource files had to be moved as well. I think patch 002 is fine.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Would something like this work? There is literally no difference between hadoop-kms.jar and hadoop-kms-classes.jar, and the former can be used as a dependency just fine.

          Show
          sjlee0 Sangjin Lee added a comment - Would something like this work? There is literally no difference between hadoop-kms.jar and hadoop-kms-classes.jar, and the former can be used as a dependency just fine.
          Hide
          jzhuge John Zhuge added a comment -

          Maybe a cleaner solution is to move MiniKMS from test to main.

          Show
          jzhuge John Zhuge added a comment - Maybe a cleaner solution is to move MiniKMS from test to main.
          Hide
          jzhuge John Zhuge added a comment -

          Sangjin Lee Couldn't find out a better approach than patch 001, probably because hadoop-hdfs:test depends on hadoop-kms:test which in turn depends on hadoop-kms:main. Not easy to set up this kind of transitive dependencies. But I am not Maven expect, someone might have a good solution?

          Show
          jzhuge John Zhuge added a comment - Sangjin Lee Couldn't find out a better approach than patch 001, probably because hadoop-hdfs:test depends on hadoop-kms:test which in turn depends on hadoop-kms:main. Not easy to set up this kind of transitive dependencies. But I am not Maven expect, someone might have a good solution?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 14m 34s trunk passed
          +1 compile 10m 7s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 javadoc 0m 17s trunk passed
          +1 mvninstall 0m 14s the patch passed
          +1 compile 9m 32s the patch passed
          +1 javac 9m 32s the patch passed
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 0m 17s the patch passed
          +1 unit 2m 14s hadoop-kms in the patch passed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          41m 25s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13961
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846475/HADOOP-13961.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 8d141d8bb620 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 945db55
          Default Java 1.8.0_111
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11405/testReport/
          modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11405/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 14m 34s trunk passed +1 compile 10m 7s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 16s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 14s the patch passed +1 compile 9m 32s the patch passed +1 javac 9m 32s the patch passed +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 17s the patch passed +1 unit 2m 14s hadoop-kms in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 41m 25s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13961 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846475/HADOOP-13961.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 8d141d8bb620 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 945db55 Default Java 1.8.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11405/testReport/ modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11405/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          That is what I thought initially. Maybe hadoop-hdfs pom should be updated. Let me give it a try.

          Show
          jzhuge John Zhuge added a comment - That is what I thought initially. Maybe hadoop-hdfs pom should be updated. Let me give it a try.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks for the patch John Zhuge!

          Quick question: now that we've switched from war to jar for kms, do we need a separate classes.jar any more? Can hadoop-hdfs simply depend on the main kms jar? Is the main jar redundant with the classes.jar or are they different so that hadoop-hdfs should depend on classes.jar still?

          Show
          sjlee0 Sangjin Lee added a comment - Thanks for the patch John Zhuge ! Quick question: now that we've switched from war to jar for kms, do we need a separate classes.jar any more? Can hadoop-hdfs simply depend on the main kms jar? Is the main jar redundant with the classes.jar or are they different so that hadoop-hdfs should depend on classes.jar still?
          Hide
          jzhuge John Zhuge added a comment -

          Patch 001

          • Restore the maven-jar-plugin sections removed accidentally by HADOOP-13597.

          Testing #1 existing source tree

          1. rm -fr ~/.m2/repository/
          2. ( cd hadoop-maven-plugins/ && mvn install )
          3. mvn clean
          4. mvn install -DskipTests -Dmaven.javadoc.skip -Pnative

          Testing #2 new source tree

          1. rm -fr ~/.m2/repository/
          2. mvn install -DskipTests -Dmaven.javadoc.skip -Pnative

          Should have run both test cases for any major pom change.

          Show
          jzhuge John Zhuge added a comment - Patch 001 Restore the maven-jar-plugin sections removed accidentally by HADOOP-13597 . Testing #1 existing source tree rm -fr ~/.m2/repository/ ( cd hadoop-maven-plugins/ && mvn install ) mvn clean mvn install -DskipTests -Dmaven.javadoc.skip -Pnative Testing #2 new source tree rm -fr ~/.m2/repository/ mvn install -DskipTests -Dmaven.javadoc.skip -Pnative Should have run both test cases for any major pom change.
          Hide
          jzhuge John Zhuge added a comment -

          Thanks Sangjin Lee. I have reproduced the issue with just mvn install and will fix it.

          Show
          jzhuge John Zhuge added a comment - Thanks Sangjin Lee . I have reproduced the issue with just mvn install and will fix it.
          Hide
          sjlee0 Sangjin Lee added a comment -

          I don't know about the hadoop-common issue John Zhuge is seeing, but I can reproduce the original problem pretty easily. With a clean trunk and an empty maven local cache the build fails:

          [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve dependencies for project org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: The following artifacts could not be resolved: org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-SNAPSHOT, org.apache.hadoop:hadoop-kms:jar:tests:3.0.0-alpha2-SNAPSHOT: Could not find artifact org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-SNAPSHOT in apache.snapshots.https (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]

          And the problem does go away if I revert HADOOP-13597.

          IMO the cause seems pretty clear. With HADOOP-13597, it appears that we stopped installing hadoop-kms-classes.jr and hadoop-kms-tests.jar (https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-kms/3.0.0-alpha2-SNAPSHOT/). But there are projects that depend on those classifiers (hadoop-hdfs being one).

          Show
          sjlee0 Sangjin Lee added a comment - I don't know about the hadoop-common issue John Zhuge is seeing, but I can reproduce the original problem pretty easily. With a clean trunk and an empty maven local cache the build fails: [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve dependencies for project org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: The following artifacts could not be resolved: org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-SNAPSHOT, org.apache.hadoop:hadoop-kms:jar:tests:3.0.0-alpha2-SNAPSHOT: Could not find artifact org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-SNAPSHOT in apache.snapshots.https ( https://repository.apache.org/content/repositories/snapshots ) -> [Help 1] And the problem does go away if I revert HADOOP-13597 . IMO the cause seems pretty clear. With HADOOP-13597 , it appears that we stopped installing hadoop-kms-classes.jr and hadoop-kms-tests.jar ( https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-kms/3.0.0-alpha2-SNAPSHOT/ ). But there are projects that depend on those classifiers (hadoop-hdfs being one).
          Hide
          aw Allen Wittenauer added a comment -

          Note that the Hadoop-trunk-Commit is having issues (on certain nodes?) which directly impacts whats in the nexus.

          Show
          aw Allen Wittenauer added a comment - Note that the Hadoop-trunk-Commit is having issues (on certain nodes?) which directly impacts whats in the nexus.
          Hide
          aw Allen Wittenauer added a comment -

          That's a normal failure due to the weird way the Hadoop build works. You can't clean without doing an install first since hadoop-maven-plugins is built on the fly and not pushed to nexus repo.

          Show
          aw Allen Wittenauer added a comment - That's a normal failure due to the weird way the Hadoop build works. You can't clean without doing an install first since hadoop-maven-plugins is built on the fly and not pushed to nexus repo.
          Hide
          jzhuge John Zhuge added a comment - - edited

          Reverting HADOOP-13597 does not fix my mvn clean issue.

          Show
          jzhuge John Zhuge added a comment - - edited Reverting HADOOP-13597 does not fix my mvn clean issue.
          Hide
          jzhuge John Zhuge added a comment -

          My tree sync'd to:

          f59e36b YARN-6068. Log aggregation get failed when NM restart even with recovery (Junping Du via Varun Saxena)
          
          Show
          jzhuge John Zhuge added a comment - My tree sync'd to: f59e36b YARN-6068. Log aggregation get failed when NM restart even with recovery (Junping Du via Varun Saxena)
          Hide
          jzhuge John Zhuge added a comment - - edited

          Got something really strange when trying to reproduce Karthik's issue:

          • rm -fr ~/.m2/*, deleting everything including settings.xml
          • git clean -f -d

          mvn clean failed in hadoop-common. It has not reached hadoop-kms yet.

          [ERROR] Plugin org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT
           or one of its dependencies could not be resolved: Could not find artifact 
          org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT -> [Help 1]
          
          Show
          jzhuge John Zhuge added a comment - - edited Got something really strange when trying to reproduce Karthik's issue: rm -fr ~/.m2/*, deleting everything including settings.xml git clean -f -d mvn clean failed in hadoop-common . It has not reached hadoop-kms yet. [ERROR] Plugin org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT -> [Help 1]
          Hide
          sjlee0 Sangjin Lee added a comment -

          Could it be related with HADOOP-13597?

          Show
          sjlee0 Sangjin Lee added a comment - Could it be related with HADOOP-13597 ?

            People

            • Assignee:
              andrew.wang Andrew Wang
              Reporter:
              kasha Karthik Kambatla
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development