We could eliminate the vast majority of the manual instructions if we require everyone to

• build locally
• use the docker image to make the build

                        if [[ "${bundle.snappy.in.bin}" == "true" ]]; then
                          if [[ "${bundle.snappy}" == "true" ]]; then
                            cd "${snappy.lib}"
                            $$TAR *snappy* | (cd $${TARGET_BIN_DIR}/; $$UNTAR)
                            if [[ $? -ne 0 ]]; then
                              echo "Bundling snappy bin files failed"
                              exit 1
                            fi
                          fi

Ping Chris Nauroth. I have a question about the logic here...

Why does this cd to snappy.lib and copy the contents into bin when all those bits should have already been copied as part of the bundle.snappy into the lib directory? This seems to imply that on Windows (which appears to be the only place this is used), the libraries have been copied to two different directories...

Allen Wittenauer, yes, this is Windows-specific logic. Windows wants DLLs to be in bin instead of the standard lib directory structure. On Windows, the native build artifacts get placed into a bin sub-directory as part of the hadoop-common build. The hadoop-common pom.xml also sets bundle.snappy.in.bin to true, so that later the distro knows snappy.dll needs to go here.

This seems to imply that on Windows (which appears to be the only place this is used), the libraries have been copied to two different directories...

I don't think so, because there are -d checks to trigger the copies only if the directories exist. On Windows, LIB_DIR will not exist and BIN_DIR will exist, so it will only do the copy for BIN_DIR. On non-Windows, LIB_DIR will exist and BIN_DIR will not exist, so it will only do the copy for LIB_DIR.

BTW, it looks like lifting out the dist-layout-stitching script to an external file on trunk isn't working for Windows. I filed HADOOP-12899, and I'll take a closer look.

OK, that kind of makes sense now. I missed that Windows wasn't setting LIB_DIR. I'll throw some comments in the code so that others aren't confused by it. Thanks!

I'll throw some comments in the code so that others aren't confused by it.

Great idea. Much appreciated.

-00:

• dev-support/create-release.sh renamed to dev-support/bin/create-release
• add docker support
• native is now optional
• add more native build options to match docker image
• replace changes and release notes handling to use yetus
• add support for gpg signing
• add some files to .gitignore
• detect which jvm is in use in the pom and use that version in docker mode (this fixes a bug with mvn site:stage when using certain versions of JDK8?)
• set a location for log output and artifacts output to make the screen have a bit more sanity when running interactively
• pull lots of shell code out of poms
• properly parameterize the shell code rather than be dependent upon maven replacement
• massive shell code cleanup
• upgrade yetus to 0.2.0 so that rdm works as expected
• properly set defaults for various native lib vars

TODO:

• custom maven cache, esp important when building on jenkins since there is a race condition otherwise (which likely means our builds are very suspect)
• set custom hadoop version
• test bundling
• test on windows

-1 overall

This message was automatically generated.

Hi Allen, thanks for working on this. I gave this a try on my machine, and it worked except for needing to change /usr/bin/bash to /bin/bash in the two scripts. Or should we be using /usr/bin/env to find bash instead?

Some other misc review comments:

• "yetus_abs" doesn't seem yetus specific, rename to something more generic?
• set_defaults_part1 does some grepping to determine maven variables, can also do that with the help plugin:

-> % mvn -f hadoop-project/pom.xml help:evaluate -Dexpression=javac.version | grep -v "\["
1.7
-> % mvn help:evaluate -Dexpression=project.version | grep -v "\["
3.0.0-SNAPSHOT

• What does the -Papache-release profile do? I don't see it mentioned in our pom.xml's, and it at the end it always prints the "don't forget to sign" message. If this is generating signatures, then it'd be good to adjust the success message to also indicate the paths.
• typo for imgname in dockermode: "createrelase" -> "createrelease"

These comments are all pretty minor though, do you think there is anything important feature-wise that is still missing? If not, we could check this in after a rev and do the rest as follow-on.

Thanks for the review!

it worked except for needing to change /usr/bin/bash to /bin/bash in the two scripts. Or should we be using /usr/bin/env to find bash instead?

I'll modify to use env.

"yetus_abs" doesn't seem yetus specific, rename to something more generic?

Sure. (At some point, I'm planning on making all of Hadoop's shell code dependent upon the yetus bash library so that I've got all the code in one place... especially since most of the performance enhancements will likely go there first since it's easier for me to get code committed there.)

set_defaults_part1 does some grepping to determine maven variables, can also do that with the help plugin:

I thought about doing it that way, but the bash way is significantly faster due to no JVM overhead. But I can change it if people want.

What does the -Papache-release profile do?

It's part of the maven-release-plugin. I popped it in there as I want to try and use it to generate signed jars, fix version numbers, etc during the build process. That's a work in progress.

If this is generating signatures, then it'd be good to adjust the success message to also indicate the paths.

apache-release (or using --sign option) should be triggering a signing phase. Agreed that the script should probably remind the user where --artifactsdir is located.

typo for imgname in dockermode: "createrelase" -> "createrelease"

Easy fix.

do you think there is anything important feature-wise that is still missing?

Yes. I absolutely must fix the maven cache problem. Any sort of parallel maven builds have race conditions. This means that there is a potential of incorrect release artifacts since Hadoop switched to maven and almost with 100% certainty since a) Hadoop release instructions moved to building on Jenkins and b) parallel builds were enabled on Jenkins in late 2014. (This means that releases made since about 2.6.0 are highly suspect. Whee!) One of the key reasons I added the docker support was to make it easier to force RMs into a pseudo-protected customized environment. Now I just need to complete that work.

Great, thanks for the updates!

I thought about doing it that way, but the bash way is significantly faster due to no JVM overhead. But I can change it if people want.

Dropping a comment about the help plugin is sufficient. The build takes about 15 mins though, so not sure a couple seconds of savings are a big deal.

Yes. I absolutely must fix the maven cache problem.

Regarding the maven cache, is this at all addressed by the release builds not being SNAPSHOT versions? All the precommit stuff should be SNAPSHOT, and I doubt there are multiple RMs building the same non-SNAPSHOT release version.

Otherwise, is the fix specifying a fresh maven.repo.local?

Happy to help out however, I'm gonna start trying 3.0 builds as soon as this goes in.

Regarding the maven cache, is this at all addressed by the release builds not being SNAPSHOT versions? All the precommit stuff should be SNAPSHOT, and I doubt there are multiple RMs building the same non-SNAPSHOT release version.

I'll get to this in a second, but be aware there is still a process problem here:

a) RM fires off a job on a shared machine to make a release
b) Someone submits a patch that changes hadoop's or some other project's build such that it either mistakenly nukes the .m2 directory or maliciously changes the contents of jars or other blackhat stuff to other running processes on the Jenkins server
c) RM takes release artifacts, signs them and then puts them up for vote, completely unaware of what is actually in the package...

In other words, doing releases on the Jenkins slaves is an absolutely terrible idea. Docker is not going to protect us here. (See also http://www.apache.org/dev/release.html#owned-controlled-hardware, which means if we're not breaking the letter of the law, we're definitely breaking the spirit...)

Now that we've eliminated running this on jenkins for anything real, we're back to RMs running this on their own boxes. RM who are much more likely to be running things in parallel, even on the same branch....

the apache-release profile is usually defined in the apache parent pom (ex: the profile in org.apache:apache:17).

It's usually set up to help make sure projects are generating valid release artifacts. It looks like Hadoop doesn't use the ASF parent pom though?

That was my assessment too.

Hadoop's release process is ... not good. I've been (slowly) gearing up to make a run at a 3.x release for while now. Every corner I turn seems to be littered with more crud that needs to be cleaned up and manual processes that make no sense.

(... and yes, the rewrite of the release notes builder 2-3 years ago is when I started on this journey... sigh.)

So, is the fix specifying an independent m2 dir? This means we end up downloading the world, but I've seen some good improvements by bumping the # of download threads and using Google's maven mirror.

So, is the fix specifying an independent m2 dir?

With all the wrappings that need to go into making that work effectively, yes. That at least solves the maven cache problem.

This means we end up downloading the world

Let's not forget that this is for a release. Downloading the world is a good thing.

BTW, don't forget that HADOOP-12893 still needs to be taken care of as well.

I've been digging into our release artifacts today while testing my latest iteration of this and found some concerning things...

From what I can tell, there are effectively 4 ways in which bits can appear in the final dist:

• Use the dist-layout-stitching script (either in hadoop-dist/pom.xml in branch-2 or pulled out into dev-support in trunk)
• Custom handling in hadoop-assemblies
• Custom handling in that modules's pom.xml
• Custom handling elsewhere that needs to be dug out (e.g., hadoop-mapreduce-project specifically handling the stuff in hadoop-mapreduce-client-nativetask)

If a module is added that doesn't do any of these things, then they effectively disappear. For trunk, I've noticed two oddities:

• hdfs-client source, test, and dependency jars are missing
• hadoop-hdfs-native-client completely disappears

There are likely more.

-01:

• stupid hacks to make hdfs-client* somewhat bundle better (dist-layout-stitching, pom component, cmake changes, etc)
• maven cache support
• better signing support
• --asfrelease option now sets everything for actually making a release
• quite a few bug fixes
• took out the .gitignore changes
• sets JAVA_HOME to the JVM version in the maven file when running Docker
• when in --asfrelease mode, make sure the signing key is in the KEYS file

-1 overall

This message was automatically generated.

Thanks for the rev Allen, sorry for not getting to this sooner. Some review comments:

• Typo "relase" in help text and the docker imgname
• Shall we change "patchprocess" to "releaseprocess" or some such for LOGDIR?
• -asfrelease sets NATIVE and SIGN, so the "-asfrelease requires --sign" validation will never trigger
• Should we even support building on Darwin? I'd rather it just abort for unsupported platforms. Is windows supported for that matter?
• The docker run command will pass along --dockercache, which then prints a spurious error msg:

$ docker run -i -t --privileged -v /home/andrew/.gnupg:/home/andrew/.gnupg -v /home/andrew/dev/apache/trunk/patchprocess:/home/andrew/dev/apache/trunk/patchprocess -v /home/andrew/dev/apache/trunk/target/artifacts:/home/andrew/dev/apache/trunk/target/artifacts -v /home/andrew/dev/apache/trunk:/build/source -u andrew -w /build/source hadoop/createrelase:3.0.0-SNAPSHOT_norc /build/source/dev-support/bin/create-release --mvncache=/maven --asfrelease --dockercache --indocker
ERROR: docker mode not enabled. Disabling dockercache.

• DOCKERAN isn't set anywhere, remove this check?
• MVNCACHE is ignored when --dockercache is specified and not --indocker right? That would be a good validation.
• Nit: some extra newlines at the end of makearelease
• When I did --asfrelease with --docker, it hung without output waiting for my GPG passphrase. I typed in my PW a few times, but it kept prompting, possibly because there wasn't a GPG agent running. Ended up Ctrl-D'ing it.
• Related to previous, it'd be nice to be able to do --sign independent of the build. Means we should filter out ".asc" and ".mds" files in signartifacts in case it's invoked twice though. I moved signartifacts out of its if statement as a test, which worked.

Thanks for the review. Clearly some bugs. But some clarification/comments:

Shall we change "patchprocess" to "releaseprocess" or some such for LOGDIR?

patchprocess is in .gitignore and who knows where else and I didn't want to add another one to reduce the amount of pollution.

--asfrelease sets NATIVE and SIGN, so the "--asfrelease requires --sign" validation will never trigger

Actually, it can:

  if [[ ! -d "${HOME}/.gnupg" ]]; then
    hadoop_error "ERROR: No .gnupg dir. Disabling signing capability."
    SIGN=false
  fi
...
  if [[ "${ASFRELEASE}" = true ]]; then
    if [[ "${SIGN}" = false ]]; then
      hadoop_error "ERROR: --asfrelease requires --sign. Exiting."
      exit
    fi

In other words, if there is no .gnupg directory, then override the user's request for signing support as a warning. But if an ASF release is requested, fail out.

Should we even support building on Darwin? I'd rather it just abort for unsupported platforms. Is windows supported for that matter?

There's no reason why someone can't use create-release to build non-Apache releases...

Related to previous, it'd be nice to be able to do --sign independent of the build.

IMO, but that's an outside of the Apache process and outside of create-release problem. I don't think I'd trust an RM that can't write a simple shell for loop. One of my goals here is to lock in stone the ASF release process since it's pretty clear that people screw it up all the time and/or do very bad things (like build them on Jenkins).

Thanks Allen, sounds reasonable overall. Looking forward to the rev.

In other words, if there is no .gnupg directory, then override the user's request for signing support as a warning. But if an ASF release is requested, fail out.

Missed that, gotcha. Maybe instead of ERROR we say WARNING when we aren't exiting, since ERRORs are typically fatal.

There's no reason why someone can't use create-release to build non-Apache releases...

Not sure who is using create-release to build non-ASF releases. Anyway, perhaps we only do OS enforcement when running with asfrelease? I think this fits with "lock in stone" as you say.

-02:

• a ton of fixes
• need to add some bits to the dockerfile for gpg
• reorganized dockerfile to make it easier to add stuff

A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9161/console in case of problems.

-1 overall

This message was automatically generated.

-03:

• clean up some output
• fix hdfs.h

A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9165/console in case of problems.

-1 overall

This message was automatically generated.

Thanks for the rev Allen, LGTM +1. Tested locally and seems to have worked. I see you chose not to do some of the option validations I recommended, but that's a matter of taste.

Only nit is that there's an extra space in --asfrelease usage text. Minor enough that we can fix on at commit time? IDK if you have anything else feature-wise you want to add.

Thanks for the review.

There are somethings I wanted to fix yet (e.g., fuse), but I can do those later.

Committed to trunk.

FAILURE: Integrated in Hadoop-trunk-Commit #9681 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9681/)
HADOOP-12892. fix/rewrite create-release (aw) (aw: rev 7b1c37a13a55dc184f1b64439b9928b53e352ee7)

• dev-support/create-release.sh
• dev-support/bin/dist-layout-stitching
• dev-support/bin/dist-copynativelibs
• hadoop-hdfs-project/hadoop-hdfs-native-client/src/CMakeLists.txt
• hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
• dev-support/docker/Dockerfile
• hadoop-project-dist/pom.xml
• hadoop-hdfs-project/hadoop-hdfs-client/pom.xml
• hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
• dev-support/bin/create-release

After this in..hadoop-hdfs-native-client compilation failing by throwing the following...

F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: F:\GitCode\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\target\bin\RelWithDebInfo does not exist.
around Ant part ...<copy todir="F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target/bin">... @ 14:98 in F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:317)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:152)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:555)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:158)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\target\bin\RelWithDebInfo does not exist.
around Ant part ...<copy todir="F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target/bin">... @ 14:98 in F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
        at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:355)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:106)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
        ... 19 more

IIUC, following fix should be fine,, didn't run in Linux environment....Andrew Wang and Allen Wittenauer if you people agree, will raise separate Jira and upload the patch..Thanks..

@@ -148,7 +148,7 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd">
                     </exec>
                     <!-- Copy for inclusion in distribution. -->
                     <copy todir="${project.build.directory}/bin">
-                      <fileset dir="${project.build.directory}/native/target/bin/RelWithDebInfo"/>
+                      <fileset dir="${project.build.directory}/native/bin/RelWithDebInfo"/>

Commit date given as " Date: 05-03-2016 05:12:04 ". I think, it should be April-27th,2016.

Revision: 7b1c37a13a55dc184f1b64439b9928b53e352ee7
Author: Allen Wittenauer <aw@apache.org>
Date: 05-03-2016 05:12:04
Message:
HADOOP-12892. fix/rewrite create-release (aw)

that is the author date, and presumably when Allen first committed work related to this fix. If you look at the full commit you'll see the commit date is in fact on the 27th of April:

$ git log --format=fuller --grep HADOOP-12892
commit 7b1c37a13a55dc184f1b64439b9928b53e352ee7
Author:     Allen Wittenauer <aw@apache.org>
AuthorDate: Fri Mar 4 15:42:04 2016 -0800
Commit:     Allen Wittenauer <aw@apache.org>
CommitDate: Wed Apr 27 08:38:22 2016 -0700

    HADOOP-12892. fix/rewrite create-release (aw)

And the canonical commit date you should look at is the message to the common-commits@hadoop mailing list:

https://s.apache.org/hadoop-commit-7b1c37a13a55dc184f1b64439b9928b53e352ee7

Sean Busbey thanks for calrification..

I thought based on commit date, git log will be generated ( even in the Jira, Filed : Development ) which makes little confusion atleast to me..

$ git log --grep HADOOP-12892
commit 7b1c37a13a55dc184f1b64439b9928b53e352ee7
Author: Allen Wittenauer <aw@apache.org>
Date:   Fri Mar 4 15:42:04 2016 -0800

    HADOOP-12892. fix/rewrite create-release (aw)

bytheway I came to know --format=fuller now..

Raised HDFS-10353 to fix this compilation error in windows..

I'm reopening this since we need this in branch-2.8 to handle the new releasedocs generation. I made a quick cherry-pick attempt and the ISA-L stuff at least needs to be removed.

I discovered this broke the Windows distro build, and I'm filing a patch on HADOOP-13149 to fix it.

Andrew Wang, this is marked to be a incompatible change, do you know why it is marked to that? How can we back port an incompatible change to 2.8?

Thanks,

I think Allen was the one who flagged it as incompatible (so he can correct me if I'm wrong), but my understanding is that it's only incompatible for people who are building releases. This might affect Bigtop or distribution vendors, but in the end it should be producing the same src and bin tarballs.

So I think it's safe to backport to 2.8, and probably doesn't need to be flagged as incompatible since it doesn't affect end users.

Thanks Andrew Wang, sorry for my late response, I missed your comment above.

I took a quick try to backport this patch to branch-2.8, it has a couple of conflicts. At least, it depends on HADOOP-12850, and HADOOP-12850 depends on more commits like HADOOP-10115, which again are marked as an incompatible change.

Given a couple of incompatible dependencies of this patch, I felt risky to backport all of them to branch-2.8. Since create-release works in branch-2 before branch-2.8, do you know which JIRA introduced failures of create-release in branch-2? Can we revert problematic commits which leads to the failure?

So one missing feature in the branch-2/branch-2.8 create-release script is that it doesn't use the new -Preleasedocs to build the CHANGES.txt file. I'm not sure how much work it is to add this support, but that's the alternative if we don't backport this JIRA.

Could you provide some more context on what breaks if we don't include HADOOP-12850 and so on? It might be less work to tweak the backport to avoid these dependent JIRAs compared to adding Preleasedocs support to the old script.

Just a reminder that the old method of building on Jenkins violates ASF policy. Also:

• I need to update the HowToRelease docs to use create-release
• -Pdocs is missing from the build, just like it was in the original create-release.sh file.

If -PreleaseDocs is relatively easy to be fixed in branch-2/branch-2.8, we should look at it. Allen Wittenauer, could you add your points regarding to which JIRA we need to backport to branch-2 to support -Preleasedocs? I can try to backport them.

Since I have very limited understanding of shell scripts, I don't really know what will be broken if we don't include HADOOP-12850.

Is the create-release in branch-2 broken by HADOOP-11792?

Sorry, branch-2 is not a priority of my volunteer time.

I've updated HowToRelease based upon having dev-support/bin/create-release and fixing quite a few things that were missing (jdiff!) or wrong (people.apache.org!). Also note that I filed HADOOP-13245 to fix up a few more things in the source and add some functionality.

Attaching a patch for branch-2.

Difference between 03 patch and 01.branch-2 patch

• 03 patch fixes dist-layout-stitching but branch-2 doesn't have the file. The file was created by HADOOP-12850 but I didn't cherry-pick the commit because HADOOP-12850 is an incompatible change. I fixed hadoop-dist/pom.xml instead.
• Removed ISA-L related code.

A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9692/console in case of problems.

-1 overall

This message was automatically generated.

Backport LGTM +1, thanks Akira. I'll assume you did appropriate local testing (running create-release with some different args). Will let you confirm and commit.

Thanks Akira Ajisaka working on this fix.

I just tried to run the create-release locally (OSX) without any parameters. It fails by following errors:

****************************************************************************
                            Staging the release
****************************************************************************


$ mv /Users/wtan/sandbox/hadoop/target/staging/hadoop-project /Users/wtan/sandbox/hadoop/target/r2.9.0-SNAPSHOT/
$ cd /Users/wtan/sandbox/hadoop/target/
$ tar czpf hadoop-site-2.9.0-SNAPSHOT.tar.gz r2.9.0-SNAPSHOT/api r2.9.0-SNAPSHOT/css r2.9.0-SNAPSHOT/dependency-analysis.html r2.9.0-SNAPSHOT/hadoop-annotations r2.9.0-SNAPSHOT/hadoop-ant r2.9.0-SNAPSHOT/hadoop-archive-logs r2.9.0-SNAPSHOT/hadoop-archives r2.9.0-SNAPSHOT/hadoop-assemblies r2.9.0-SNAPSHOT/hadoop-auth r2.9.0-SNAPSHOT/hadoop-auth-examples r2.9.0-SNAPSHOT/hadoop-aws r2.9.0-SNAPSHOT/hadoop-azure r2.9.0-SNAPSHOT/hadoop-common-project r2.9.0-SNAPSHOT/hadoop-datajoin r2.9.0-SNAPSHOT/hadoop-dist r2.9.0-SNAPSHOT/hadoop-distcp r2.9.0-SNAPSHOT/hadoop-extras r2.9.0-SNAPSHOT/hadoop-gridmix r2.9.0-SNAPSHOT/hadoop-hdfs-bkjournal r2.9.0-SNAPSHOT/hadoop-hdfs-httpfs r2.9.0-SNAPSHOT/hadoop-hdfs-nfs r2.9.0-SNAPSHOT/hadoop-hdfs-project r2.9.0-SNAPSHOT/hadoop-kms r2.9.0-SNAPSHOT/hadoop-mapreduce r2.9.0-SNAPSHOT/hadoop-mapreduce-client r2.9.0-SNAPSHOT/hadoop-mapreduce-examples r2.9.0-SNAPSHOT/hadoop-maven-plugins r2.9.0-SNAPSHOT/hadoop-minicluster r2.9.0-SNAPSHOT/hadoop-minikdc r2.9.0-SNAPSHOT/hadoop-nfs r2.9.0-SNAPSHOT/hadoop-openstack r2.9.0-SNAPSHOT/hadoop-pipes r2.9.0-SNAPSHOT/hadoop-project-dist r2.9.0-SNAPSHOT/hadoop-rumen r2.9.0-SNAPSHOT/hadoop-sls r2.9.0-SNAPSHOT/hadoop-streaming r2.9.0-SNAPSHOT/hadoop-tools r2.9.0-SNAPSHOT/hadoop-yarn r2.9.0-SNAPSHOT/hadoop-yarn-project r2.9.0-SNAPSHOT/images r2.9.0-SNAPSHOT/index.html r2.9.0-SNAPSHOT/project-reports.html
$ cd /Users/wtan/sandbox/hadoop
$ cp -p /Users/wtan/sandbox/hadoop/hadoop-common-project/hadoop-common/src/site/markdown/release/2.9.0-SNAPSHOT/CHANGES*.md /Users/wtan/sandbox/hadoop/target/artifacts/CHANGES.md
cp: /Users/wtan/sandbox/hadoop/hadoop-common-project/hadoop-common/src/site/markdown/release/2.9.0-SNAPSHOT/CHANGES*.md: No such file or directory

Failed!

Could you let me know which parameter you are using to do tests locally?

Thanks,

I hadn't tested locally, and I faced the same error. I'll look into the cause.

I noticed

• We need to set the version to 2.9.0 before running create-release in branch-2
• 2.9.0/CHANGES*.md should be in ${basedir}/src/site/markdown/release but it is in patchprocess/. It seems that the workingDirectory does not work.
  hadoop-common-project/hadoop-common/pom.xml

                          <executable>${basedir}/../../dev-support/bin/releasedocmaker</executable>
                          <workingDirectory>src/site/markdown/release/</workingDirectory>
  

yetus-wrapper changes the working directory to patchprocess/ and execute the script specified by the first argument, so the workingDirectory does not work.

pushd "${HADOOP_PATCHPROCESS}" >/dev/null
if [[ $? != 0 ]]; then
  yetus_error "ERROR: yetus-dl: Cannot pushd to ${HADOOP_PATCHPROCESS}"
  exit 1
fi

We need to copy 2.9.0/CHANGES*.md from patchprocess/.

Updated the patch. Ran create-release successfully on my MBA.

A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9702/console in case of problems.

02.branch-2 patch:

• Removed workingDirectory
• Copy CHANGES*.md from patchprocess instead of workingDirectory
• Create the destination directory before copying documents into the directory.

FYI: create-release --asfrelease option requires --docker option and the option requires HADOOP-13245.

-1 overall

This message was automatically generated.

yetus-wrapper changes the working directory to patchprocess/ and execute the script specified by the first argument, so the workingDirectory does not work.

It changes it back just prior to execution.

(Although, now that releasedocmaker supports a location in yetus 0.3.0, we should probably change the pom.xml to use it.)

It changes it back just prior to execution.

Thank you for the reply. This is true after HADOOP-12743, which it is not in branch-2. I'll backport this.

03.branch-2 patch: After backporting HADOOP-12743, reverted the followings

• Removed workingDirectory
• Copy CHANGES*.md from patchprocess instead of workingDirectory

A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9706/console in case of problems.

-1 overall

This message was automatically generated.

Thanks Akira Ajisaka!

I've tried to run create-release on my local box (without --docker) on branch-2.

It succeeded and can generate artifacts. I've manually take a look at generated sites, and start a simple YARN cluster runs few distributed shell job, all succeeded.

The patch has few conflicts while applying to branch-2.8. Could you update patch for branch-2.8? I can help with verification.

Rebased the patch for branch-2.8. I couldn't verified the patch because of the connection failure in releasedocmaker. I filed YETUS-419 and created a patch.

A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9720/console in case of problems.

-1 overall

This message was automatically generated.

Akira Ajisaka, Thanks for rebasing the patch.

I have tried this patch on branch-2.8, and artifacts can be built locally, I have also verified sites and launch YARN pseudo cluster using built binaries.

For the fix itself, I will +1 since the problem is already fixed. Not sure about shell check and shell doc -1, should we treat it like an error or just a warning.

I would like to get someone more familiar with this field to review and commit the patch.
cc: Andrew Wang.

Hi Andrew Wang, would you review the latest patches? (03.branch-2 and 01.branch-2.8)

Took a quick look, LGTM thanks Akira, also thanks Wangda for validating.

Forgot to mention, I looked at the shellcheck/shell doc errors too, these exist in the trunk version too. I'm not sure why they weren't picked up by the original precommit run.

Committed to branch-2 and branch-2.8. Thanks Wangda Tan and Andrew Wang for review!

After backporting this issue, we need to backport HDFS-10353, HADOOP-13149, and HADOOP-13245 as well. Let's do this.