Details
-
Type:
Improvement
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 3.0.0, trunk-win
-
Fix Version/s: 2.1.0-beta
-
Component/s: build
-
Labels:None
-
Target Version/s:
-
Hadoop Flags:Reviewed
Description
Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
Issue Links
- breaks
-
HADOOP-9249
hadoop-maven-plugins version-info goal causes build failure when running with Clover
-
- Resolved
-
-
HADOOP-9260
Hadoop version may be not correct when starting name node or data node
-
- Closed
-
-
HADOOP-9245
mvn clean without running mvn install before fails
-
- Closed
-
-
HADOOP-9315
Port HADOOP-9249 hadoop-maven-plugins Clover fix to branch-2 to fix build failures
-
- Closed
-
-
HADOOP-9279
Document the need to build hadoop-maven-plugins for eclipse and separate project builds
-
- Closed
-
- depends upon
-
HADOOP-9116
test and source JARs should not be in cluster classpath
-
- Patch Available
-
- is depended upon by
-
HADOOP-9117
replace protoc ant plugin exec with a maven plugin
-
- Closed
-
- is related to
-
HADOOP-9207
version info source checksum does not include all source files
-
- Open
-
-
HADOOP-8420
Hadoop Common creating package-info.java must not depend on sh
-
- Resolved
-
-
HADOOP-9251
mvn eclipse:eclipse fails on trunk
-
- Resolved
-
- is required by
-
YARN-158
Yarn creating package-info.java must not depend on sh
-
- Resolved
-
- relates to
-
HADOOP-9082
Simplify scripting usages so that parallel platform-dependent scripts are simple to maintain
-
- Resolved
-
- requires
-
HADOOP-9202
test-patch.sh fails during mvn eclipse:eclipse if patch adds a new module to the build
-
- Resolved
-
Activity
Hi, Colin. Sorry to hear that this patch caused trouble in your dev environment. There had been a related patch, HADOOP-9202, to change test-patch.sh to run mvn install before checking mvn eclipse:eclipse. My understanding is that the established process is to run mvn install first to guarantee that everything is deployed into your local Maven repository, and then run mvn eclipse:eclipse. That's based on the instructions shown here:
http://wiki.apache.org/hadoop/EclipseEnvironment
From this directory you just 'cd'-ed into (Which is also known as the top-level directory of a branch or a trunk checkout), perform:
$ mvn install -DskipTests
$ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
Does running an mvn install first fix the issue for you? I just tested that this works in my environment by running:
rm -rf ~/.m2/repository/org/apache/hadoop/hadoop-maven-plugins && mvn install -DskipTests && mvn eclipse:eclipse
(The rm -rf was to force removal of any cached copy of the plugin that I might have in my local repository.)
It seems like this broke mvn eclipse:eclipse.
I get this output:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.478s
[INFO] Finished at: Mon Jan 21 18:30:18 PST 2013
[INFO] Final Memory: 33M/661M
[INFO] ------------------------------------------------------------------------
[ERROR] Plugin org.apache.hadoop:hadoop-maven-plugins:3.0.0-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-SNAPSHOT -> [Help 1]
org.apache.maven.plugin.PluginResolutionException: Plugin org.apache.hadoop:hadoop-maven-plugins:3.0.0-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-SNAPSHOT
at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:140)
at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getPluginDescriptor(DefaultMavenPluginManager. java:142)
at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getMojoDescriptor(DefaultMavenPluginManager.java: 261)
at org.apache.maven.plugin.DefaultBuildPluginManager.getMojoDescriptor(DefaultBuildPluginManager.java:185)
at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator. calculateLifecycleMappings(DefaultLifecycleExecutionPlanCalculator.java:280)
at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator. calculateForkedLifecycle(DefaultLifecycleExecutionPlanCalculator.java:520)
The underlying issue seems to be that generate-resources is broken, and maven-eclipse-plugin "Invokes the execution of the lifecycle phase generate-resources prior to executing itself." (As described here: http://maven.apache.org/plugins/maven-eclipse-plugin/eclipse-mojo.html )
You can see that this command also fails with the same error: mvn generate-resources. In fact, a lot of the early phases of the default lifecycle seem to fail: validate, initialize, generate-sources. For some reason mvn site works, though, despite the fact that it should be implicitly calling those earlier build phases.
Integrated in Hadoop-Mapreduce-trunk #1320 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1320/)
HADOOP-8924. Add CHANGES.txt description missed in commit r1435380. (Revision 1436181)
Result = SUCCESS
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1436181
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
Integrated in Hadoop-Hdfs-trunk #1292 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1292/)
HADOOP-8924. Add CHANGES.txt description missed in commit r1435380. (Revision 1436181)
Result = FAILURE
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1436181
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
Integrated in Hadoop-Yarn-trunk #103 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/103/)
HADOOP-8924. Add CHANGES.txt description missed in commit r1435380. (Revision 1436181)
Result = SUCCESS
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1436181
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
Thank you, Suresh. I am attaching a branch-2 patch. The differences compared to the trunk patch are:
- hadoop-common-project/hadoop-common/pom.xml had diverged on branch-2, so I needed to do a (trivial) manual resolution.
- hadoop-maven-plugins/pom.xml references 2.0.3-SNAPSHOT as its parent version instead of 3.0.0-SNAPSHOT
Integrated in Hadoop-trunk-Commit #3264 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3264/)
HADOOP-8924. Add CHANGES.txt description missed in commit r1435380. (Revision 1436181)
Result = SUCCESS
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1436181
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
I committed the patch to trunk and branch-trunk-win. Thank you Chris and Alejandro.
BTW I could not get the merge to build in branch-2. Chris, please upload a branch-2 patch or let me know what I am missing in the merge, if you want it to be merged to branch-2.
Integrated in Hadoop-Mapreduce-trunk #1318 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1318/)
HADOOP-8924. Add maven plugin alternative to shell script to save package-info.java. Contributed by Alejandro Abdelnur and Chris Nauroth. (Revision 1435380)
HADOOP-8924. Revert r1435372 that missed some files (Revision 1435379)
HADOOP-8924. Add maven plugin alternative to shell script to save package-info.java. Contributed by Alejandro Abdelnur and Chris Nauroth. (Revision 1435372)
Result = SUCCESS
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435380
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/saveVersion.sh
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/HadoopVersionAnnotation.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionInfo.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/common-version-info.properties
- /hadoop/common/trunk/hadoop-maven-plugins
- /hadoop/common/trunk/hadoop-maven-plugins/pom.xml
- /hadoop/common/trunk/hadoop-maven-plugins/src
- /hadoop/common/trunk/hadoop-maven-plugins/src/main
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util/Exec.java
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util/FileSetUtils.java
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/versioninfo
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/versioninfo/VersionInfoMojo.java
- /hadoop/common/trunk/hadoop-project/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/scripts/saveVersion.sh
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/YarnVersionInfo.java
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-version-info.properties
- /hadoop/common/trunk/pom.xml
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435379
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/saveVersion.sh
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/HadoopVersionAnnotation.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionInfo.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/common-version-info.properties
- /hadoop/common/trunk/hadoop-project/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/scripts/saveVersion.sh
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/YarnVersionInfo.java
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-version-info.properties
- /hadoop/common/trunk/pom.xml
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435372
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/saveVersion.sh
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/HadoopVersionAnnotation.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionInfo.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/common-version-info.properties
- /hadoop/common/trunk/hadoop-project/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/scripts/saveVersion.sh
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/YarnVersionInfo.java
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-version-info.properties
- /hadoop/common/trunk/pom.xml
Integrated in Hadoop-Hdfs-trunk #1290 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1290/)
HADOOP-8924. Add maven plugin alternative to shell script to save package-info.java. Contributed by Alejandro Abdelnur and Chris Nauroth. (Revision 1435380)
HADOOP-8924. Revert r1435372 that missed some files (Revision 1435379)
HADOOP-8924. Add maven plugin alternative to shell script to save package-info.java. Contributed by Alejandro Abdelnur and Chris Nauroth. (Revision 1435372)
Result = FAILURE
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435380
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/saveVersion.sh
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/HadoopVersionAnnotation.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionInfo.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/common-version-info.properties
- /hadoop/common/trunk/hadoop-maven-plugins
- /hadoop/common/trunk/hadoop-maven-plugins/pom.xml
- /hadoop/common/trunk/hadoop-maven-plugins/src
- /hadoop/common/trunk/hadoop-maven-plugins/src/main
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util/Exec.java
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util/FileSetUtils.java
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/versioninfo
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/versioninfo/VersionInfoMojo.java
- /hadoop/common/trunk/hadoop-project/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/scripts/saveVersion.sh
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/YarnVersionInfo.java
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-version-info.properties
- /hadoop/common/trunk/pom.xml
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435379
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/saveVersion.sh
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/HadoopVersionAnnotation.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionInfo.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/common-version-info.properties
- /hadoop/common/trunk/hadoop-project/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/scripts/saveVersion.sh
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/YarnVersionInfo.java
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-version-info.properties
- /hadoop/common/trunk/pom.xml
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435372
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/saveVersion.sh
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/HadoopVersionAnnotation.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionInfo.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/common-version-info.properties
- /hadoop/common/trunk/hadoop-project/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/scripts/saveVersion.sh
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/YarnVersionInfo.java
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-version-info.properties
- /hadoop/common/trunk/pom.xml
Integrated in Hadoop-Yarn-trunk #101 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/101/)
HADOOP-8924. Add maven plugin alternative to shell script to save package-info.java. Contributed by Alejandro Abdelnur and Chris Nauroth. (Revision 1435380)
HADOOP-8924. Revert r1435372 that missed some files (Revision 1435379)
HADOOP-8924. Add maven plugin alternative to shell script to save package-info.java. Contributed by Alejandro Abdelnur and Chris Nauroth. (Revision 1435372)
Result = SUCCESS
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435380
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/saveVersion.sh
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/HadoopVersionAnnotation.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionInfo.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/common-version-info.properties
- /hadoop/common/trunk/hadoop-maven-plugins
- /hadoop/common/trunk/hadoop-maven-plugins/pom.xml
- /hadoop/common/trunk/hadoop-maven-plugins/src
- /hadoop/common/trunk/hadoop-maven-plugins/src/main
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util/Exec.java
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util/FileSetUtils.java
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/versioninfo
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/versioninfo/VersionInfoMojo.java
- /hadoop/common/trunk/hadoop-project/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/scripts/saveVersion.sh
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/YarnVersionInfo.java
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-version-info.properties
- /hadoop/common/trunk/pom.xml
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435379
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/saveVersion.sh
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/HadoopVersionAnnotation.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionInfo.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/common-version-info.properties
- /hadoop/common/trunk/hadoop-project/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/scripts/saveVersion.sh
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/YarnVersionInfo.java
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-version-info.properties
- /hadoop/common/trunk/pom.xml
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435372
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/saveVersion.sh
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/HadoopVersionAnnotation.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionInfo.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/common-version-info.properties
- /hadoop/common/trunk/hadoop-project/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/scripts/saveVersion.sh
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/YarnVersionInfo.java
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-version-info.properties
- /hadoop/common/trunk/pom.xml
Integrated in Hadoop-trunk-Commit #3263 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3263/)
HADOOP-8924. Add maven plugin alternative to shell script to save package-info.java. Contributed by Alejandro Abdelnur and Chris Nauroth. (Revision 1435380)
HADOOP-8924. Revert r1435372 that missed some files (Revision 1435379)
Result = SUCCESS
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435380
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/saveVersion.sh
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/HadoopVersionAnnotation.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionInfo.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/common-version-info.properties
- /hadoop/common/trunk/hadoop-maven-plugins
- /hadoop/common/trunk/hadoop-maven-plugins/pom.xml
- /hadoop/common/trunk/hadoop-maven-plugins/src
- /hadoop/common/trunk/hadoop-maven-plugins/src/main
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util/Exec.java
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util/FileSetUtils.java
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/versioninfo
- /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/versioninfo/VersionInfoMojo.java
- /hadoop/common/trunk/hadoop-project/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/scripts/saveVersion.sh
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/YarnVersionInfo.java
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-version-info.properties
- /hadoop/common/trunk/pom.xml
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435379
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/saveVersion.sh
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/HadoopVersionAnnotation.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionInfo.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/common-version-info.properties
- /hadoop/common/trunk/hadoop-project/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/scripts/saveVersion.sh
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/YarnVersionInfo.java
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-version-info.properties
- /hadoop/common/trunk/pom.xml
Integrated in Hadoop-trunk-Commit #3262 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3262/)
HADOOP-8924. Add maven plugin alternative to shell script to save package-info.java. Contributed by Alejandro Abdelnur and Chris Nauroth. (Revision 1435372)
Result = FAILURE
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435372
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/saveVersion.sh
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/HadoopVersionAnnotation.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionInfo.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/common-version-info.properties
- /hadoop/common/trunk/hadoop-project/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/scripts/saveVersion.sh
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/YarnVersionInfo.java
- /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-version-info.properties
- /hadoop/common/trunk/pom.xml
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12565540/HADOOP-8924.7.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-maven-plugins hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2072//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2072//console
This message is automatically generated.
+1 for the trunk patch. I will follow HADOOP-9207 to ensure the comments I made earlier is handled.
Attaching version 7 of the patch for trunk and branch-trunk-win. This adds the pom.xml comments and more javadocs for the Maven plugin classes.
Sorry, I missed Suresh's comment about javadocs before uploading that last patch. Thanks for the feedback, Suresh. I'll do that now.
Attaching version 6 of the patch. The only difference from the prior version is that I added comments to pom.xml for hadoop-common and hadoop-yarn-common explaining the resource exclusion/inclusion with property substitution.
Yes, we can simplify the VersionInfo.java if we get rid of the YARN one. Regarding the exclude/include of the props file in the POM, is to do copy with filtering.
We can consolidate to a single VersionInfo class and delete YarnVersionInfo in the scope of HADOOP-9207, which will convert the build to calculate a single checksum across the whole repository.
I'll upload a new patch shortly with the comment added to pom.xml.
One other comment - please add javadoc for the newly added classes. For example, it is missing for Excec, FileSetUtils etc.
Looking at the code VersionInfo.java code looks like it can be improved. It has _get* instance methods and get* static methods. These static methods are hidden by other classes that extend this class. Is it not possible to use one implementation of VersionInfo? All that I see different between variants of this class is how they decide on the properties file to read. So instead of passing component name in the constructor, can the properties name be passed?
In pom.xml file, the change where in resources *-version-info.properties seems to be both included and excluded. Adding a brief comment on why this is being done will help understanding.
Finally, I think we should have an additional goal in print-version-info that prints the computed MD5, this would help somebody to easily obtain the checksum of a source and verify a build.
Chris, latest patch LGTM, but again, somebody else should review it as I've written part of it. Thx. Also, I've just commented in HADOOP-9207 how we could do the checksum for ALL source.
Thanks, Matt. I just created HADOOP-9207 to address it separately.
The most important use of the checksum is to enforce the constraint that all the servers talking to each other be running code compiled from the same source tree.
Considering this, I think the best solution is to calculate a single global checksum for the whole project and use the same value across all modules. That's going to take some restructuring, and I'd prefer to address it in a separate jira instead of holding up this one.
Alejandro, I think this is all set as long as you are OK with the version 5 patch and my justifications on the tests and release audit warnings. Let me know what you think.
Hi Chris and Alejandro, it's fine with me to test on Mac, Win, and Ubuntu, for consistent MD5, and I agree it isn't strictly necessary to have the same MD5 as saveVersion.sh.
But I'm concerned about:
One thing I've forgot to mention is that currently (and with this patch) MD5 are done only for the sources in common and in yarn. And the VersionInfo from common is used in hdfs. IMO, we should either have a global MD5 & VersionInfo for the whole project or one per module. This is out of scope of this JIRA, just wanted to bring it up.
I didn't notice because I was focusing on the Hadoop-1 version where I'm more familiar with the env. In Hadoop-1 there is simply one checksum for the whole project.
Absent that, I think the checksum for hdfs in Hadoop-2 should be created by summing the MD5 for the hdfs sub-project sources, so each sub-project sums its own sources (as do common and yarn). The most important use of the checksum is to enforce the constraint that all the servers talking to each other be running code compiled from the same source tree. And that is clearly important to HDFS, and won't be enforced with the current scheme.
Agree if this behavior was already in the Hadoop-2 code we don't have to fix it here, but if it would be a simple change, I would support fixing it here. If not, please open a bug for it. Thanks.
The release audit failures are unrelated to this patch:
!????? /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/hdfsdatanodes.odg !????? /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/hdfsarchitecture.odg Lines that start with ????? in the release audit report indicate files that do not have an Apache license header.
It seems like rat can't figure out that these are binary files. I haven't been able to repro the problem on any of my own machines. This seems to be causing a problem for some other patches too.
The failure in TestZKFailoverController is unrelated. This test has been flaky lately, failing on a few other patches.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12564737/HADOOP-8924.5.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
-1 release audit. The applied patch generated 2 release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-maven-plugins hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:
org.apache.hadoop.ha.TestZKFailoverController
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2040//testReport/
Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/2040//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2040//console
This message is automatically generated.
I'm attaching version 5 of the patch to rebase it against recent trunk changes.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12564579/HADOOP-8924.4.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2039//console
This message is automatically generated.
HADOOP-9202 just got committed. (Thank you, Robert Joseph Evans.) I'm resubmitting the v4 patch to Jenkins.
I'm attaching version 4 of the patch. Jenkins reported new javadoc warnings. This was caused by the non-standard doclet tags like @goal used by Maven. To resolve this, I've switched the code to use Maven's plugin annotations instead of doclet tags.
Additionally, Jenkins reported a failure for this patch during mvn eclipse:eclipse. This is actually a bug in test-patch.sh. I've submitted a separate patch to address that in HADOOP-9202. The HADOOP-9202 patch must be committed first before we can commit HADOOP-8924, so I would appreciate a code review on that one too. It's a small diff that just reorders some of the build steps.
Jenkins also gave -1 for no new tests. I think this is acceptable for a piece of build scripting, but if you disagree, let me know, and I can work on some tests.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12564490/HADOOP-8924.3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 javadoc. The javadoc tool appears to have generated 2 warning messages.
-1 eclipse:eclipse. The patch failed to build with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-maven-plugins hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:
org.apache.hadoop.ha.TestZKFailoverController
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2030//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2030//console
This message is automatically generated.
patch LGTM (pending Jenkins), but somebody else should +1 as I'm partially responsible for it.
One thing I've forgot to mention is that currently (and with this patch) MD5 are done only for the sources in common and in yarn. And the VersionInfo from common is used in hdfs. IMO, we should either have a global MD5 & VersionInfo for the whole project or one per module. This is out of scope of this JIRA, just wanted to bring it up.
Re-uploading trunk patch so that Jenkins picks up the right file.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12564482/HADOOP-8924-branch-trunk-win.3.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2029//console
This message is automatically generated.
I'm attaching version 3 of the patch for trunk and branch-trunk-win. The differences since last time are that *.proto is included in the fileset for the MD5 calculation (see pom.xml files), and the MD5 calculation uses a platform-independent sort order for processing the files (see VersionInfoMojo#computeMD5). The sort logic is a port of the earlier Python code on branch-trunk-win.
I've tested this on Mac, Windows, and Ubuntu. MD5 calculations are consistent across platforms (though intentionally different from saveVersion.sh for the reasons discussed above).
Line endings on the .proto files were consistent across platforms. The .java files generated by protoc had Windows line endings though.
I'll add *.proto to the fileset in my next version of the patch.
on generated files, I assume you refer to the protoc files, we could add the protoc sources the MD5.
Thanks, Matt and Alejandro. I'm seeing differences between MD5s on Mac vs. Windows due to file sorting differences. I'm working on an updated patch to address it.
One other difference between this and saveVersion.sh is that saveVersion.sh included generated-sources, but this plugin won't, because we're binding it to the initialize phase. During the Python port, we chose to exclude generated-sources, because we can't guarantee that generated stuff (i.e. protoc) will generate code with the same line endings regardless of platform. This had been causing different MD5s on Windows, so we changed the Python script to skip generated-sources. I think skipping generated-sources is fine. I'm just mentioning it here so that everyone is aware of the difference.
Matt, I was planning on testing Mac, Windows, and Ubuntu, because that's what I have access to right now. Is there any particular reason to retest on RHEL5 and RHEL6 in addition to that? That probably would have been important for the Python script, because RHEL distributions are tightly coupled to a particular Python version, but I don't think it's relevant for the Java implementation. If you disagree, let me know, and I'll spin up some RHEL VMs.
Yeah, I'm good with this approach.
The MD5 computed by saveVersion.sh is:
srcChecksum=`find src/main/java -name '*.java' | LC_ALL=C sort | xargs md5sum | md5sum | cut -d ' ' -f 1`
This is, sort the files, MD5 on each file, and them MD5 on the MD5 output of all files.
The MD5 computed by the plugin is a single MD5 on the content of ALL files, sorted.
So the MD5s computed by the script and the plugin won't be the same.
But the MD5 of the plugin should always be the same in diff platforms for the same source.
+1, conditional on those tests, and test-patch results.
Yes, this looks okay, and thanks for doing this, Chris. The "real work" is being done in Java [ack Alejandro for the first version of that], which is goodness and also now a transparent part of the Hadoop distribution. And invoking it from a Maven plugin is also [I'll admit :-)] goodness, and hopefully makes this approach acceptable to you, Alejandro?
Chris, please test against RHEL5, RHEL6, and Windows. Hopefully the MD5's will be the same for all three platforms, and the same as the saveVersion.sh script produces. (Achieving those constraints was the tricky part of the python version.) Thanks.
Chris, thanks for taking up on this. LGTM, submitting patch for test-patch to pick it up.
Matt, is this variant of the approach OK with you? If so, would you withdraw your -1?
The discussion on common-dev mentioned the idea of using a Maven plugin, but building it within the reactor as a new sub-module of the main Hadoop project instead of a separately deployed artifact. It was mentioned that Avro already does this kind of thing.
I wanted to see what this would look like, so I took the Maven plugin code from Alejandro's github repo, copied it into the hadoop-common tree, and adjusted the pom.xml files to reference this. (I did not include ProtocMojo.java, since that isn't the scope of this jira.) I'm attaching patches for trunk and branch-trunk-win that show this work.
I'm hopeful that this approach will be an acceptable compromise for everyone. The version info code is still part of the main Hadoop project, easily visible to any Hadoop developer or release engineer. The Maven plugin framework does require some extra boilerplate code, but the focus is still on generating the version info. By making it a plugin instead of a stand-alone executable, we avoid the need to duplicate the code for common and yarn, like Radim mentioned.
My solution will end with 2 copies of that code in hadoop tree. Without writing hadoop plugin or using scripting language, i do not know better way.
it might be workable. Store saveVersion.java into separate directory but same module as source files, then bind maven-compiler and maven-exec plugins to generate-sources or process-sources phase.
How about the following:
1. convert saveVersion.sh into saveVersion.java. Pass the parameter values (including the root of the source tree for checksum calculation) in to the class constructor at invocation time.
2. add a dependency rule that causes saveVersion.class to be compiled before any other java is compiled, and separately from it.
3. add a simple execution outcall from ant or maven that calls the class, and passes the parameter values in to the call. If maven requires a completely vanilla "plug-in" to do this well, I'm okay with that. I just don't want the significant functionality of saveVersion.java to be subsumed into the plug-in and become opaque to the hadoop source.
I voted for Groovy or some other language in pom.xml, but Matt is -1 on groovy in maven.
My understanding of 2nd vote option was to use an existing maven plugin and combine it with some language which JVM provides. There is couple of options available.
I offered to rework python code into Groovy in maven, because its high priority for me to get builds working on windows. I can rework it into pretty much anything just you guys need to decide what direction you want to go.
If I'm not mistaken the votes for using Maven plugins didn't have a single -1s, including a +1 from you Randim.
Writing Maven plugins is more complex that writing scripts, I don't dispute this. The main motivations for using Maven plugins is to keep things in the POM declarative and to hide (if necessary) different handlings for different platforms.
IMO, having a bunch of stand alone java programs that do this will have the same complexity than a Maven plugin, plus we'll lose all the benefits from Maven, version management, distribution/download via maven repo, declarative builds.
Regarding HADOOP-8887, it has been sitting around for a while, not because it is incomplete or complex but because we are not solving how the plugin gets into a Maven repo so it is available for the Hadoop build just to download it. I thinking we should have a discussion in common-dev@ on how to address this.
Going back to this particular JIRA, the other approach is to do it in the way Oozie does it:
For building a release tarball, there is a script 'mkdistro' which computes the SCM (SVN or GIT) info (uri, branch, commit), the build timestamp and passes it the Maven invocation as -D options, then they are avail as properties in the POM and the version-info.properties file gets filtered with those.
Essentially is this patch minus the plugin plus a script to call Maven.
The drawback of this approach is that the script has to do very much what saveVersion.sh does, meaning we are at square one.
We never tackled this in Oozie as a plugin because, so far, nobody as attempted to build/run Oozie server in Windows. I assume in Oozie we'd go the plugin way when such need arises.
I'm OK in updating this patch to do things how Oozie does it today, using bash. For how to do that in Windows command without using cygwin, I don't know. Again, I think we are back to square one with this approach, we are just the script logic of saveVersion.sh around.
I also do not like custom maven plugin. Can't it be reworked into separate java program?
-1. Alejandro, I've said in several threads that this level of complexity is not acceptable for this task.
We have no need to use maven plugins everywhere they could possibly be applied, when a simpler approach will do. Very few people in the community are expert with these plug-ins, and this approach is increasing the opacity, not the simplicity. HADOOP-8887 is a reasonable attempt to do something already complex (native builds) in a way that fits better into the build process. But HADOOP-8887 has been in process for two months, and apparently is still not ready for prime time. Capturing the version information should be simple, and this approach isn't simple at all. Furthermore, the underlying plugin (referenced from github) is 3 files with 374 source loc, not simple and transparent.
Along similar lines, I've created and put a patch for protoc using a maven plugin, HADOOP-9117.
Thanks Chris.
(not setting as patch avail because the plugin is not avail in a Maven repo, details follow).
Attached is patch that uses a maven plugin to generate the version information.
The patch changes the VersionInfo class to read all the info from a properties file instead of the package-info.java and the HadoopVersionInfoAnnotation.
The HadoopVersionInfoAnnotation.java and the saveVersion.sh files are deleted.
A new properties file 'common-version-info.properties' with ${} variables is available in src/main/resources:
version=${pom.version}
revision=${version-info.scm.commit}
branch=${version-info.scm.branch}
user=${user.name}
date=${version-info.build.time}
url=${version-info.scm.uri}
srcChecksum=${version-info.source.md5}
This file is copied into target using filtering and the variables are resolved at copy time.
All the variables that start with version-info are resolved and injected into the project by the new plugin hadoop-maven-plugins using the goal version-info:
...
<plugin>
<groupId>org.apache.hadoop.maven.plugin</groupId>
<artifactId>hadoop-maven-plugins</artifactId>
<executions>
<execution>
<id>version-info</id>
<goals>
<goal>version-info</goal>
</goals>
<configuration>
<source>
<directory>${basedir}/src/main/java</directory>
<includes>
<include>**/*.java</include>
</includes>
</source>
</configuration>
</execution>
</executions>
</plugin>
...
Because the plugin is not yet available, to test it you'll have to build and install it locally. The plugin source is avail at:
https://github.com/tucu00/hadoop-maven-plugins
After cloning it, running 'mvn install' will install it locally.
You'll need to apply HADOOP-9116 before this patch in order to get things to work otherwise the sources JAR ends up in the classpath with the unresolved properties files and it may be first in the classpath.
I've tested in OSX and Windows and seems to be working as expected.
I still need to verify the MD5 generated is correct.
I think the current way VersionInfo works (only getting the md5 of common is not correct), but this is out of scope from this JIRA. Also, the YARN variant seems not be used from the command line when doing 'yarn version'.
Regarding the maven plugins, we have to figure how we'll build/publish them to a Maven repo in order to use them. I personally think they should be a sub-project of Hadoop to allow us to release new ones independent of Hadoop. Why a Hadoop sub-project and not a general project? IMO, the focus of these plugins (including HADOOP-8887 for cmake and the protoc that is already avail in the GH repo, another JIRA coming soon) is to solve Hadoop build problems.
Thanks, Alejandro. I've reassigned it to you. I can volunteer to code review it when the patch is available.
Chris, As I've mentioned in the aliases earlier today, I've got a Maven plugin taking care of saveVersion.sh, I've tested in Windows, OSX, Linux. Would you mind reassigning this JIRA to me and I'll put a patch?
Thx
On branch-trunk-win, when we converted to Python, I consolidated to a single copy of saveVersion.py in /dev-support, which is used by both sub-modules. I needed to introduce some additional arguments to the script so that it could be parameterized differently for hadoop-common vs. hadoop-yarn-common. It didn't add much complexity, and it eliminated the need to maintain 2 different copies of the same code. We could take the same approach to parameterize the equivalent Java code (or whatever else).
After discussion in common-dev@, the community consensus rejected use of Python. (see references in HADOOP-9082) Also, it was encouraged to take functionality out of scripts and move it into core code.
I suggest that we convert saveVersion.sh to Java, and change the ant script to a recursive call that compiles the java class and then invokes it. Same thing should work in trunk, but be aware there are two instances of saveVersion.sh in trunk: hadoop-common-project/hadoop-common/dev-support/saveVersion.sh and hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/scripts/saveVersion.sh. They are slightly different, but I suspect that can and should be subsumed.
Its agreed to merge this into trunk? I can prepare and test version of this patch.
Related jira from branch-1-win
Hi Matt, a discussion on common-dev@ makes sense to me. I don't feel strongly about what the right answer is, it just wasn't obvious to me that it's python. If it's determined to be python, then that's fine by me.
And yes, whenever this is resolved, we should definitely file a JIRA to update BUILDING.txt accordingly.
Hi Aaron, I advised Chris to use Python. As a general matter I think we should stay cross-platform wherever it makes sense to do so. In other words, if something is obviously totally platform-dependent, then go ahead and do two versions, one in shell for Linux, and one in powershell or cmd for Windows. However, where it can be platform-independent, I think we should use a platform-independent scripting language to write a single script (which may include some conditional code for platform-dependent cases if necessary). Obviously it works fine in this case.
Python is only one possibility, of course. However it seemed a reasonable choice. It's object-oriented, with reasonable IDE support. It's free (like beer and speech), available on essentially all platforms, and in good odour with the OSS community. The only other evident candidate would be Ruby, but I think more people know Python than Ruby, altho I can't substantiate that. (Altho there is the Tiobe Index, which I don't claim is authoritative.)
This is worth discussing, because as we integrate the Windows port, we'll have lots of opportunity to follow whatever model we agree on. Shall we continue here, or pop it to a common-dev thread?
And I agree with you that whatever answer we agree on should be documented as a build dependency.
Hey guys, does it really make sense to trade a dependency on sh for a dependency on python? Maybe it does, but at least on Unix systems I feel like sh is more likely to be available than python. (Honest question here - not trying to be a pain.)
At the very least, if we stick with this being in python, we should update BUILDING.txt to say that we now have a dependency on python (and perhaps some specific version of python?) in order to build Hadoop.
I committed the patch to branch-trunk-win. Thank you Chris.
+1 for the patch. I will commit it to branch-trunk-win.
This patch converts saveVersion.sh to an equivalent saveVersion.py. I also expanded the template and the arguments to include additional fields, so that the same script can be used from both the Hadoop Common and Yarn modules.
I have tested "mvn generate-sources" in the following combinations:
1. Mac/git
2. Windows/git
3. Windows + Cygwin/git
4. Mac/svn
5. Windows/svn
6. Windows + Cygwin/svn
All tests produced the correct package-info.java, with the same value for srcChecksum.
I'm starting this change in branch-trunk-win, because the driver for this change is to get a working Windows build for a branch off of trunk.
Hi Chris,
Running the install first works for me. It's not an ideal solution, but I guess we can live with it, as long as it's documented. Hopefully future versions of Maven will offer some way out of this chicken and egg problem with bundled Maven plugins.
Now that we have a place to put Maven plugins, I guess it's time to reopen HADOOP-8887.
By the way, thanks for doing this work.