Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
2.7.3, 2.6.5
-
None
-
-
Reviewed
Description
After HADOOP-12893, we are seeing mvn install -DskipTests failing in branch-2.7, branch-2.7.3, and branch-2.6. This failure is caused by the followings
- hadoop-project module depends on hadoop-build-tools module, but hadoop-project module does not declare hadoop-build-tools as its submodule. Therefore, hadoop-build-tools is not built before building hadoop-project.
- hadoop-build-tools pom and jar are not uploaded to the snapshot repository (https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/)
The build failure occurs if the both of the above conditions are satisfied.
Attachments
Attachments
- HADOOP-13297.00.patch
- 3 kB
- Akira Ajisaka
- HADOOP-13297.branch-2.6.00.patch
- 3 kB
- Akira Ajisaka
- HADOOP-13297.1.patch
- 1.0 kB
- Sean Busbey
- HADOOP-13297.branch-2.6.v1.patch
- 1.0 kB
- Sean Busbey
Issue Links
- is broken by
-
HADOOP-12893 Verify LICENSE.txt and NOTICE.txt
- Closed
Activity
I no longer have sufficient perms on the HADOOP tracker to self-assign, but I plan to take a try at this issue this week.
Hi busbey, I added you into contributors role in Hadoop Common and assigned.
BTW, there's also an email about the L&N files got copied into hadoop-build-tools/src/main/resources/META-INF/ , could you please help to have that covered in this too busbey?
If you think that should go with a separate jira, feel free to let me know and I'll create one. Thanks!
please file a different jira, since that issue is likely going to require different changes.
I thought there was a work around that allowed the build to work with the incorrect ordering? presuming there is, then I would not consider this a blocker. (but I am not a PMC member)
Hmm, thank you for the clarification. Marking this as blocker since this can lead build failure.
busbey how is your progress?
I had not started yet. If you'd like to take over, I'd be happy to help review or answer questions as they come up.
Attaching a patch for workaround.
- Remove the dependency to hadoop-build-tools from hadoop-common
- Add the dependency to the root module.
That moves the need for the hadoop-build-tools module even earlier in the build and doesn't actually add it as a dependency.
Is this still a blocker? I thought there was a work-around for 2.7.3 already?
That moves the need for the hadoop-build-tools module even earlier in the build
Yes. hadoop-build-tools is a submodule of the root module, so when the root module needs hadoop-build-tools module, build the module. That way the failure does not occur.
Is this still a blocker? I thought there was a work-around for 2.7.3 already?
I don't think this is still a blocker. A work-around is to push the artifact to the snapshot repository for each branch.
2.6.5-SNAPSHOT artifact is not pushed, so we can test in branch-2.6.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 14m 38s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
0 | mvndep | 0m 24s | Maven dependency ordering for branch |
-1 | mvninstall | 0m 29s | root in branch-2.6 failed. |
-1 | compile | 3m 33s | root in branch-2.6 failed with JDK v1.8.0_91. |
-1 | compile | 1m 49s | root in branch-2.6 failed with JDK v1.7.0_101. |
-1 | mvnsite | 0m 11s | root in branch-2.6 failed. |
-1 | mvneclipse | 0m 14s | root in branch-2.6 failed. |
-1 | javadoc | 0m 12s | root in branch-2.6 failed with JDK v1.8.0_91. |
-1 | javadoc | 0m 11s | root in branch-2.6 failed with JDK v1.7.0_101. |
0 | mvndep | 0m 7s | Maven dependency ordering for patch |
+1 | mvninstall | 6m 32s | the patch passed |
-1 | compile | 1m 38s | root in the patch failed with JDK v1.8.0_91. |
-1 | javac | 1m 38s | root in the patch failed with JDK v1.8.0_91. |
-1 | compile | 1m 46s | root in the patch failed with JDK v1.7.0_101. |
-1 | javac | 1m 46s | root in the patch failed with JDK v1.7.0_101. |
+1 | mvnsite | 7m 46s | the patch passed |
+1 | mvneclipse | 1m 23s | the patch passed |
-1 | whitespace | 0m 0s | The patch has 934 line(s) that end in whitespace. Use git apply --whitespace=fix. |
-1 | whitespace | 0m 24s | The patch 22 line(s) with tabs. |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
-1 | javadoc | 0m 13s | root in the patch failed with JDK v1.8.0_91. |
-1 | javadoc | 7m 39s | root-jdk1.7.0_101 with JDK v1.7.0_101 generated 66 new + 0 unchanged - 0 fixed = 66 total (was 0) |
-1 | unit | 18m 44s | root in the patch failed with JDK v1.7.0_101. |
-1 | asflicense | 0m 39s | The patch generated 114 ASF License warnings. |
87m 43s |
Reason | Tests |
---|---|
JDK v1.8.0_91 Failed junit tests | hadoop.security.ssl.TestReloadingX509TrustManager |
hadoop.security.ssl.TestSSLFactory | |
hadoop.http.TestSSLHttpServer | |
hadoop.http.TestHttpCookieFlag | |
hadoop.io.TestUTF8 | |
JDK v1.7.0_101 Failed junit tests | hadoop.fs.TestS3_LocalFileContextURI |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 20s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
0 | mvndep | 0m 12s | Maven dependency ordering for branch |
+1 | mvninstall | 6m 35s | trunk passed |
+1 | compile | 7m 6s | trunk passed |
+1 | mvnsite | 8m 39s | trunk passed |
+1 | mvneclipse | 0m 47s | trunk passed |
+1 | javadoc | 4m 37s | trunk passed |
0 | mvndep | 0m 11s | Maven dependency ordering for patch |
+1 | mvninstall | 6m 26s | the patch passed |
+1 | compile | 6m 37s | the patch passed |
+1 | javac | 6m 37s | the patch passed |
+1 | mvnsite | 8m 33s | the patch passed |
+1 | mvneclipse | 0m 34s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 2s | The patch has no ill-formed XML file. |
+1 | javadoc | 4m 38s | the patch passed |
-1 | unit | 95m 10s | root in the patch failed. |
+1 | asflicense | 0m 24s | The patch does not generate ASF License warnings. |
151m 36s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:9560f25 |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12816902/HADOOP-13297.00.patch |
JIRA Issue | |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml |
uname | Linux c7552842f9fd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / c04c5ec |
Default Java | 1.8.0_91 |
unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/9946/artifact/patchprocess/patch-unit-root.txt |
Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/9946/testReport/ |
modules | C: hadoop-project . U: . |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/9946/console |
Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
The problem is that hadoop-project defines a use of the remote-resources-plugin that needs the hadoop-build-tool jar but that plugin declaration does not declare the dependency. You can check this by running mvn validate to look at the reactor order. Here you can see that the Hadoop Build Tools module is way at the end, because Maven doesn’t think anything needs it:
[INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] Apache Hadoop Main [INFO] Apache Hadoop Project POM [INFO] Apache Hadoop Annotations [INFO] Apache Hadoop Project Dist POM [INFO] Apache Hadoop Assemblies [INFO] Apache Hadoop Maven Plugins [INFO] Apache Hadoop MiniKDC [INFO] Apache Hadoop Auth [INFO] Apache Hadoop Auth Examples [INFO] Apache Hadoop Common [INFO] Apache Hadoop NFS [INFO] Apache Hadoop KMS [INFO] Apache Hadoop Common Project [INFO] Apache Hadoop HDFS [INFO] Apache Hadoop HttpFS [INFO] Apache Hadoop HDFS BookKeeper Journal [INFO] Apache Hadoop HDFS-NFS [INFO] Apache Hadoop HDFS Project [INFO] hadoop-yarn [INFO] hadoop-yarn-api [INFO] hadoop-yarn-common [INFO] hadoop-yarn-server [INFO] hadoop-yarn-server-common [INFO] hadoop-yarn-server-nodemanager [INFO] hadoop-yarn-server-web-proxy [INFO] hadoop-yarn-server-applicationhistoryservice [INFO] hadoop-yarn-server-resourcemanager [INFO] hadoop-yarn-server-tests [INFO] hadoop-yarn-client [INFO] hadoop-yarn-applications [INFO] hadoop-yarn-applications-distributedshell [INFO] hadoop-yarn-applications-unmanaged-am-launcher [INFO] hadoop-yarn-site [INFO] hadoop-yarn-registry [INFO] hadoop-yarn-project [INFO] hadoop-mapreduce-client [INFO] hadoop-mapreduce-client-core [INFO] hadoop-mapreduce-client-common [INFO] hadoop-mapreduce-client-shuffle [INFO] hadoop-mapreduce-client-app [INFO] hadoop-mapreduce-client-hs [INFO] hadoop-mapreduce-client-jobclient [INFO] hadoop-mapreduce-client-hs-plugins [INFO] Apache Hadoop MapReduce Examples [INFO] hadoop-mapreduce [INFO] Apache Hadoop MapReduce Streaming [INFO] Apache Hadoop Distributed Copy [INFO] Apache Hadoop Archives [INFO] Apache Hadoop Rumen [INFO] Apache Hadoop Gridmix [INFO] Apache Hadoop Data Join [INFO] Apache Hadoop Ant Tasks [INFO] Apache Hadoop Extras [INFO] Apache Hadoop Pipes [INFO] Apache Hadoop OpenStack support [INFO] Apache Hadoop Amazon Web Services support [INFO] Apache Hadoop Client [INFO] Apache Hadoop Mini-Cluster [INFO] Apache Hadoop Scheduler Load Simulator [INFO] Apache Hadoop Tools Dist [INFO] Apache Hadoop Tools [INFO] Apache Hadoop Distribution [INFO] Apache Hadoop Build Tools
after patch on branch-2.6, we can see that the build tools module happens right away, as it should:
[INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] Apache Hadoop Main [INFO] Apache Hadoop Build Tools [INFO] Apache Hadoop Project POM [INFO] Apache Hadoop Annotations [INFO] Apache Hadoop Project Dist POM [INFO] Apache Hadoop Assemblies [INFO] Apache Hadoop Maven Plugins [INFO] Apache Hadoop MiniKDC [INFO] Apache Hadoop Auth [INFO] Apache Hadoop Auth Examples [INFO] Apache Hadoop Common [INFO] Apache Hadoop NFS [INFO] Apache Hadoop KMS [INFO] Apache Hadoop Common Project [INFO] Apache Hadoop HDFS [INFO] Apache Hadoop HttpFS [INFO] Apache Hadoop HDFS BookKeeper Journal [INFO] Apache Hadoop HDFS-NFS [INFO] Apache Hadoop HDFS Project [INFO] hadoop-yarn [INFO] hadoop-yarn-api [INFO] hadoop-yarn-common [INFO] hadoop-yarn-server [INFO] hadoop-yarn-server-common [INFO] hadoop-yarn-server-nodemanager [INFO] hadoop-yarn-server-web-proxy [INFO] hadoop-yarn-server-applicationhistoryservice [INFO] hadoop-yarn-server-resourcemanager [INFO] hadoop-yarn-server-tests [INFO] hadoop-yarn-client [INFO] hadoop-yarn-applications [INFO] hadoop-yarn-applications-distributedshell [INFO] hadoop-yarn-applications-unmanaged-am-launcher [INFO] hadoop-yarn-site [INFO] hadoop-yarn-registry [INFO] hadoop-yarn-project [INFO] hadoop-mapreduce-client [INFO] hadoop-mapreduce-client-core [INFO] hadoop-mapreduce-client-common [INFO] hadoop-mapreduce-client-shuffle [INFO] hadoop-mapreduce-client-app [INFO] hadoop-mapreduce-client-hs [INFO] hadoop-mapreduce-client-jobclient [INFO] hadoop-mapreduce-client-hs-plugins [INFO] Apache Hadoop MapReduce Examples [INFO] hadoop-mapreduce [INFO] Apache Hadoop MapReduce Streaming [INFO] Apache Hadoop Distributed Copy [INFO] Apache Hadoop Archives [INFO] Apache Hadoop Rumen [INFO] Apache Hadoop Gridmix [INFO] Apache Hadoop Data Join [INFO] Apache Hadoop Ant Tasks [INFO] Apache Hadoop Extras [INFO] Apache Hadoop Pipes [INFO] Apache Hadoop OpenStack support [INFO] Apache Hadoop Amazon Web Services support [INFO] Apache Hadoop Client [INFO] Apache Hadoop Mini-Cluster [INFO] Apache Hadoop Scheduler Load Simulator [INFO] Apache Hadoop Tools Dist [INFO] Apache Hadoop Tools [INFO] Apache Hadoop Distribution
This same change should go into master and all branches that use the remote-resources-plugin, though the immediate symptom is masked on 2.8+ due to HADOOP-11889
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 11m 45s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
-1 | mvninstall | 0m 46s | root in branch-2.6 failed. |
+1 | compile | 0m 6s | branch-2.6 passed with JDK v1.8.0_91 |
+1 | compile | 0m 7s | branch-2.6 passed with JDK v1.7.0_101 |
+1 | mvnsite | 0m 17s | branch-2.6 passed |
+1 | mvneclipse | 0m 14s | branch-2.6 passed |
+1 | javadoc | 0m 8s | branch-2.6 passed with JDK v1.8.0_91 |
+1 | javadoc | 0m 7s | branch-2.6 passed with JDK v1.7.0_101 |
+1 | mvninstall | 0m 8s | the patch passed |
+1 | compile | 0m 6s | the patch passed with JDK v1.8.0_91 |
+1 | javac | 0m 6s | the patch passed |
+1 | compile | 0m 7s | the patch passed with JDK v1.7.0_101 |
+1 | javac | 0m 7s | the patch passed |
+1 | mvnsite | 0m 11s | the patch passed |
+1 | mvneclipse | 0m 8s | the patch passed |
-1 | whitespace | 0m 0s | The patch has 1131 line(s) that end in whitespace. Use git apply --whitespace=fix. |
-1 | whitespace | 0m 26s | The patch 22 line(s) with tabs. |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
+1 | javadoc | 0m 6s | the patch passed with JDK v1.8.0_91 |
+1 | javadoc | 0m 8s | the patch passed with JDK v1.7.0_101 |
+1 | unit | 0m 7s | hadoop-project in the patch passed with JDK v1.7.0_101. |
-1 | asflicense | 0m 27s | The patch generated 66 ASF License warnings. |
16m 38s |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:44eef0e |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12817274/HADOOP-13297.branch-2.6.v1.patch |
JIRA Issue | |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml |
uname | Linux 7d71905845c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | branch-2.6 / e640944 |
Default Java | 1.7.0_101 |
Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 |
mvninstall | https://builds.apache.org/job/PreCommit-HADOOP-Build/9962/artifact/patchprocess/branch-mvninstall-root.txt |
whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/9962/artifact/patchprocess/whitespace-eol.txt |
whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/9962/artifact/patchprocess/whitespace-tabs.txt |
JDK v1.7.0_101 Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/9962/testReport/ |
asflicense | https://builds.apache.org/job/PreCommit-HADOOP-Build/9962/artifact/patchprocess/patch-asflicense-problems.txt |
modules | C: hadoop-project U: hadoop-project |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/9962/console |
Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks Sean for the nice fix and educational (to me at least) trouble-shooting details!
+1 (non-binding), verified this builds good against branch-2.6 (commit tip e64094484de0e47850148d4c33ca3c189c93c01c)
Committed this to all the related branches including branch-2.7.3. Thanks busbey for the very nice fix and thanks xiaochen for the review.
SUCCESS: Integrated in Hadoop-trunk-Commit #10076 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10076/)
HADOOP-13297. Add missing dependency in setting (aajisaka: rev 7bd5d4272cd686e06c5d5fcc489b69312dacb47b)
- hadoop-project/pom.xml
Error log in branch-2.7.3.