Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: build
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      This rewrites the release process with a new dev-support/bin/create-release script. See http://wiki.apache.org/hadoop/HowToRelease for updated instructions on how to use it.

      Description

      create-release needs some major surgery.

      1. HADOOP-12892.00.patch
        31 kB
        Allen Wittenauer
      2. HADOOP-12892.01.branch-2.8.patch
        37 kB
        Akira Ajisaka
      3. HADOOP-12892.01.branch-2.patch
        39 kB
        Akira Ajisaka
      4. HADOOP-12892.01.patch
        38 kB
        Allen Wittenauer
      5. HADOOP-12892.02.branch-2.patch
        40 kB
        Akira Ajisaka
      6. HADOOP-12892.02.patch
        40 kB
        Allen Wittenauer
      7. HADOOP-12892.03.branch-2.patch
        39 kB
        Akira Ajisaka
      8. HADOOP-12892.03.patch
        41 kB
        Allen Wittenauer

        Issue Links

          Activity

          Hide
          aw Allen Wittenauer added a comment -

          We could eliminate the vast majority of the manual instructions if we require everyone to

          • build locally
          • use the docker image to make the build
          Show
          aw Allen Wittenauer added a comment - We could eliminate the vast majority of the manual instructions if we require everyone to build locally use the docker image to make the build
          Hide
          aw Allen Wittenauer added a comment -
                                  if [[ "${bundle.snappy.in.bin}" == "true" ]]; then
                                    if [[ "${bundle.snappy}" == "true" ]]; then
                                      cd "${snappy.lib}"
                                      $$TAR *snappy* | (cd $${TARGET_BIN_DIR}/; $$UNTAR)
                                      if [[ $? -ne 0 ]]; then
                                        echo "Bundling snappy bin files failed"
                                        exit 1
                                      fi
                                    fi
          

          Ping Chris Nauroth. I have a question about the logic here...

          Why does this cd to snappy.lib and copy the contents into bin when all those bits should have already been copied as part of the bundle.snappy into the lib directory? This seems to imply that on Windows (which appears to be the only place this is used), the libraries have been copied to two different directories...

          Show
          aw Allen Wittenauer added a comment - if [[ "${bundle.snappy.in.bin}" == " true " ]]; then if [[ "${bundle.snappy}" == " true " ]]; then cd "${snappy.lib}" $$TAR *snappy* | (cd $${TARGET_BIN_DIR}/; $$UNTAR) if [[ $? -ne 0 ]]; then echo "Bundling snappy bin files failed" exit 1 fi fi Ping Chris Nauroth . I have a question about the logic here... Why does this cd to snappy.lib and copy the contents into bin when all those bits should have already been copied as part of the bundle.snappy into the lib directory? This seems to imply that on Windows (which appears to be the only place this is used), the libraries have been copied to two different directories...
          Hide
          cnauroth Chris Nauroth added a comment -

          Allen Wittenauer, yes, this is Windows-specific logic. Windows wants DLLs to be in bin instead of the standard lib directory structure. On Windows, the native build artifacts get placed into a bin sub-directory as part of the hadoop-common build. The hadoop-common pom.xml also sets bundle.snappy.in.bin to true, so that later the distro knows snappy.dll needs to go here.

          This seems to imply that on Windows (which appears to be the only place this is used), the libraries have been copied to two different directories...

          I don't think so, because there are -d checks to trigger the copies only if the directories exist. On Windows, LIB_DIR will not exist and BIN_DIR will exist, so it will only do the copy for BIN_DIR. On non-Windows, LIB_DIR will exist and BIN_DIR will not exist, so it will only do the copy for LIB_DIR.

          BTW, it looks like lifting out the dist-layout-stitching script to an external file on trunk isn't working for Windows. I filed HADOOP-12899, and I'll take a closer look.

          Show
          cnauroth Chris Nauroth added a comment - Allen Wittenauer , yes, this is Windows-specific logic. Windows wants DLLs to be in bin instead of the standard lib directory structure. On Windows, the native build artifacts get placed into a bin sub-directory as part of the hadoop-common build. The hadoop-common pom.xml also sets bundle.snappy.in.bin to true, so that later the distro knows snappy.dll needs to go here. This seems to imply that on Windows (which appears to be the only place this is used), the libraries have been copied to two different directories... I don't think so, because there are -d checks to trigger the copies only if the directories exist. On Windows, LIB_DIR will not exist and BIN_DIR will exist, so it will only do the copy for BIN_DIR . On non-Windows, LIB_DIR will exist and BIN_DIR will not exist, so it will only do the copy for LIB_DIR . BTW, it looks like lifting out the dist-layout-stitching script to an external file on trunk isn't working for Windows. I filed HADOOP-12899 , and I'll take a closer look.
          Hide
          aw Allen Wittenauer added a comment -

          OK, that kind of makes sense now. I missed that Windows wasn't setting LIB_DIR. I'll throw some comments in the code so that others aren't confused by it. Thanks!

          Show
          aw Allen Wittenauer added a comment - OK, that kind of makes sense now. I missed that Windows wasn't setting LIB_DIR. I'll throw some comments in the code so that others aren't confused by it. Thanks!
          Hide
          cnauroth Chris Nauroth added a comment -

          I'll throw some comments in the code so that others aren't confused by it.

          Great idea. Much appreciated.

          Show
          cnauroth Chris Nauroth added a comment - I'll throw some comments in the code so that others aren't confused by it. Great idea. Much appreciated.
          Hide
          aw Allen Wittenauer added a comment - - edited

          -00:

          • dev-support/create-release.sh renamed to dev-support/bin/create-release
          • add docker support
          • native is now optional
          • add more native build options to match docker image
          • replace changes and release notes handling to use yetus
          • add support for gpg signing
          • add some files to .gitignore
          • detect which jvm is in use in the pom and use that version in docker mode (this fixes a bug with mvn site:stage when using certain versions of JDK8?)
          • set a location for log output and artifacts output to make the screen have a bit more sanity when running interactively
          • pull lots of shell code out of poms
          • properly parameterize the shell code rather than be dependent upon maven replacement
          • massive shell code cleanup
          • upgrade yetus to 0.2.0 so that rdm works as expected
          • properly set defaults for various native lib vars

          TODO:

          • custom maven cache, esp important when building on jenkins since there is a race condition otherwise (which likely means our builds are very suspect)
          • set custom hadoop version
          • test bundling
          • test on windows
          Show
          aw Allen Wittenauer added a comment - - edited -00: dev-support/create-release.sh renamed to dev-support/bin/create-release add docker support native is now optional add more native build options to match docker image replace changes and release notes handling to use yetus add support for gpg signing add some files to .gitignore detect which jvm is in use in the pom and use that version in docker mode (this fixes a bug with mvn site:stage when using certain versions of JDK8?) set a location for log output and artifacts output to make the screen have a bit more sanity when running interactively pull lots of shell code out of poms properly parameterize the shell code rather than be dependent upon maven replacement massive shell code cleanup upgrade yetus to 0.2.0 so that rdm works as expected properly set defaults for various native lib vars TODO: custom maven cache, esp important when building on jenkins since there is a race condition otherwise (which likely means our builds are very suspect) set custom hadoop version test bundling test on windows
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          0 shelldocs 0m 3s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 8s Maven dependency ordering for branch
          +1 mvninstall 6m 43s trunk passed
          +1 compile 5m 42s trunk passed with JDK v1.8.0_74
          +1 compile 6m 42s trunk passed with JDK v1.7.0_95
          +1 mvnsite 9m 0s trunk passed
          +1 mvneclipse 0m 43s trunk passed
          +1 javadoc 5m 16s trunk passed with JDK v1.8.0_74
          +1 javadoc 9m 17s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 6m 39s the patch passed
          +1 compile 5m 40s the patch passed with JDK v1.8.0_74
          +1 javac 5m 40s the patch passed
          +1 compile 6m 40s the patch passed with JDK v1.7.0_95
          +1 javac 6m 40s the patch passed
          +1 mvnsite 8m 59s the patch passed
          +1 mvneclipse 0m 40s the patch passed
          +1 shellcheck 0m 9s There were no new shellcheck issues.
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 javadoc 5m 24s the patch passed with JDK v1.8.0_74
          +1 javadoc 9m 21s the patch passed with JDK v1.7.0_95
          -1 unit 94m 53s root in the patch failed with JDK v1.8.0_74.
          -1 unit 12m 37s root in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 27s Patch does not generate ASF License warnings.
          197m 25s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.hdfs.TestSafeModeWithStripedFile
          JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestSafeModeWithStripedFile
            hadoop.fs.shell.find.TestName
            hadoop.fs.shell.find.TestIname
            hadoop.metrics2.impl.TestMetricsSystemImpl
            hadoop.fs.shell.find.TestPrint0
            hadoop.fs.shell.find.TestPrint



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791839/HADOOP-12892.00.patch
          JIRA Issue HADOOP-12892
          Optional Tests asflicense shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall xml
          uname Linux 3b070b13d6af 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 391da36
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          shellcheck v0.4.3
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8810/artifact/patchprocess/patch-unit-root-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8810/artifact/patchprocess/patch-unit-root-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8810/artifact/patchprocess/patch-unit-root-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8810/artifact/patchprocess/patch-unit-root-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8810/testReport/
          modules C: hadoop-project-dist . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8810/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. 0 shelldocs 0m 3s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 8s Maven dependency ordering for branch +1 mvninstall 6m 43s trunk passed +1 compile 5m 42s trunk passed with JDK v1.8.0_74 +1 compile 6m 42s trunk passed with JDK v1.7.0_95 +1 mvnsite 9m 0s trunk passed +1 mvneclipse 0m 43s trunk passed +1 javadoc 5m 16s trunk passed with JDK v1.8.0_74 +1 javadoc 9m 17s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 6m 39s the patch passed +1 compile 5m 40s the patch passed with JDK v1.8.0_74 +1 javac 5m 40s the patch passed +1 compile 6m 40s the patch passed with JDK v1.7.0_95 +1 javac 6m 40s the patch passed +1 mvnsite 8m 59s the patch passed +1 mvneclipse 0m 40s the patch passed +1 shellcheck 0m 9s There were no new shellcheck issues. +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 javadoc 5m 24s the patch passed with JDK v1.8.0_74 +1 javadoc 9m 21s the patch passed with JDK v1.7.0_95 -1 unit 94m 53s root in the patch failed with JDK v1.8.0_74. -1 unit 12m 37s root in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 197m 25s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.hdfs.TestSafeModeWithStripedFile JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.fs.shell.find.TestName   hadoop.fs.shell.find.TestIname   hadoop.metrics2.impl.TestMetricsSystemImpl   hadoop.fs.shell.find.TestPrint0   hadoop.fs.shell.find.TestPrint Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791839/HADOOP-12892.00.patch JIRA Issue HADOOP-12892 Optional Tests asflicense shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall xml uname Linux 3b070b13d6af 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 391da36 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 shellcheck v0.4.3 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8810/artifact/patchprocess/patch-unit-root-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8810/artifact/patchprocess/patch-unit-root-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8810/artifact/patchprocess/patch-unit-root-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8810/artifact/patchprocess/patch-unit-root-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8810/testReport/ modules C: hadoop-project-dist . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8810/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Allen, thanks for working on this. I gave this a try on my machine, and it worked except for needing to change /usr/bin/bash to /bin/bash in the two scripts. Or should we be using /usr/bin/env to find bash instead?

          Some other misc review comments:

          • "yetus_abs" doesn't seem yetus specific, rename to something more generic?
          • set_defaults_part1 does some grepping to determine maven variables, can also do that with the help plugin:
          -> % mvn -f hadoop-project/pom.xml help:evaluate -Dexpression=javac.version | grep -v "\["
          1.7
          -> % mvn help:evaluate -Dexpression=project.version | grep -v "\["
          3.0.0-SNAPSHOT
          
          • What does the -Papache-release profile do? I don't see it mentioned in our pom.xml's, and it at the end it always prints the "don't forget to sign" message. If this is generating signatures, then it'd be good to adjust the success message to also indicate the paths.
          • typo for imgname in dockermode: "createrelase" -> "createrelease"

          These comments are all pretty minor though, do you think there is anything important feature-wise that is still missing? If not, we could check this in after a rev and do the rest as follow-on.

          Show
          andrew.wang Andrew Wang added a comment - Hi Allen, thanks for working on this. I gave this a try on my machine, and it worked except for needing to change /usr/bin/bash to /bin/bash in the two scripts. Or should we be using /usr/bin/env to find bash instead? Some other misc review comments: "yetus_abs" doesn't seem yetus specific, rename to something more generic? set_defaults_part1 does some grepping to determine maven variables, can also do that with the help plugin: -> % mvn -f hadoop-project/pom.xml help:evaluate -Dexpression=javac.version | grep -v "\[" 1.7 -> % mvn help:evaluate -Dexpression=project.version | grep -v "\[" 3.0.0-SNAPSHOT What does the -Papache-release profile do? I don't see it mentioned in our pom.xml's, and it at the end it always prints the "don't forget to sign" message. If this is generating signatures, then it'd be good to adjust the success message to also indicate the paths. typo for imgname in dockermode: "createrelase" -> "createrelease" These comments are all pretty minor though, do you think there is anything important feature-wise that is still missing? If not, we could check this in after a rev and do the rest as follow-on.
          Hide
          aw Allen Wittenauer added a comment -

          Thanks for the review!

          it worked except for needing to change /usr/bin/bash to /bin/bash in the two scripts. Or should we be using /usr/bin/env to find bash instead?

          I'll modify to use env.

          "yetus_abs" doesn't seem yetus specific, rename to something more generic?

          Sure. (At some point, I'm planning on making all of Hadoop's shell code dependent upon the yetus bash library so that I've got all the code in one place... especially since most of the performance enhancements will likely go there first since it's easier for me to get code committed there.)

          set_defaults_part1 does some grepping to determine maven variables, can also do that with the help plugin:

          I thought about doing it that way, but the bash way is significantly faster due to no JVM overhead. But I can change it if people want.

          What does the -Papache-release profile do?

          It's part of the maven-release-plugin. I popped it in there as I want to try and use it to generate signed jars, fix version numbers, etc during the build process. That's a work in progress.

          If this is generating signatures, then it'd be good to adjust the success message to also indicate the paths.

          apache-release (or using --sign option) should be triggering a signing phase. Agreed that the script should probably remind the user where --artifactsdir is located.

          typo for imgname in dockermode: "createrelase" -> "createrelease"

          Easy fix.

          do you think there is anything important feature-wise that is still missing?

          Yes. I absolutely must fix the maven cache problem. Any sort of parallel maven builds have race conditions. This means that there is a potential of incorrect release artifacts since Hadoop switched to maven and almost with 100% certainty since a) Hadoop release instructions moved to building on Jenkins and b) parallel builds were enabled on Jenkins in late 2014. (This means that releases made since about 2.6.0 are highly suspect. Whee!) One of the key reasons I added the docker support was to make it easier to force RMs into a pseudo-protected customized environment. Now I just need to complete that work.

          Show
          aw Allen Wittenauer added a comment - Thanks for the review! it worked except for needing to change /usr/bin/bash to /bin/bash in the two scripts. Or should we be using /usr/bin/env to find bash instead? I'll modify to use env. "yetus_abs" doesn't seem yetus specific, rename to something more generic? Sure. (At some point, I'm planning on making all of Hadoop's shell code dependent upon the yetus bash library so that I've got all the code in one place... especially since most of the performance enhancements will likely go there first since it's easier for me to get code committed there.) set_defaults_part1 does some grepping to determine maven variables, can also do that with the help plugin: I thought about doing it that way, but the bash way is significantly faster due to no JVM overhead. But I can change it if people want. What does the -Papache-release profile do? It's part of the maven-release-plugin. I popped it in there as I want to try and use it to generate signed jars, fix version numbers, etc during the build process. That's a work in progress. If this is generating signatures, then it'd be good to adjust the success message to also indicate the paths. apache-release (or using --sign option) should be triggering a signing phase. Agreed that the script should probably remind the user where --artifactsdir is located. typo for imgname in dockermode: "createrelase" -> "createrelease" Easy fix. do you think there is anything important feature-wise that is still missing? Yes. I absolutely must fix the maven cache problem. Any sort of parallel maven builds have race conditions. This means that there is a potential of incorrect release artifacts since Hadoop switched to maven and almost with 100% certainty since a) Hadoop release instructions moved to building on Jenkins and b) parallel builds were enabled on Jenkins in late 2014. (This means that releases made since about 2.6.0 are highly suspect. Whee!) One of the key reasons I added the docker support was to make it easier to force RMs into a pseudo-protected customized environment. Now I just need to complete that work.
          Hide
          andrew.wang Andrew Wang added a comment -

          Great, thanks for the updates!

          I thought about doing it that way, but the bash way is significantly faster due to no JVM overhead. But I can change it if people want.

          Dropping a comment about the help plugin is sufficient. The build takes about 15 mins though, so not sure a couple seconds of savings are a big deal.

          Yes. I absolutely must fix the maven cache problem.

          Regarding the maven cache, is this at all addressed by the release builds not being SNAPSHOT versions? All the precommit stuff should be SNAPSHOT, and I doubt there are multiple RMs building the same non-SNAPSHOT release version.

          Otherwise, is the fix specifying a fresh maven.repo.local?

          Happy to help out however, I'm gonna start trying 3.0 builds as soon as this goes in.

          Show
          andrew.wang Andrew Wang added a comment - Great, thanks for the updates! I thought about doing it that way, but the bash way is significantly faster due to no JVM overhead. But I can change it if people want. Dropping a comment about the help plugin is sufficient. The build takes about 15 mins though, so not sure a couple seconds of savings are a big deal. Yes. I absolutely must fix the maven cache problem. Regarding the maven cache, is this at all addressed by the release builds not being SNAPSHOT versions? All the precommit stuff should be SNAPSHOT, and I doubt there are multiple RMs building the same non-SNAPSHOT release version. Otherwise, is the fix specifying a fresh maven.repo.local? Happy to help out however, I'm gonna start trying 3.0 builds as soon as this goes in.
          Hide
          aw Allen Wittenauer added a comment -

          Regarding the maven cache, is this at all addressed by the release builds not being SNAPSHOT versions? All the precommit stuff should be SNAPSHOT, and I doubt there are multiple RMs building the same non-SNAPSHOT release version.

          I'll get to this in a second, but be aware there is still a process problem here:

          a) RM fires off a job on a shared machine to make a release
          b) Someone submits a patch that changes hadoop's or some other project's build such that it either mistakenly nukes the .m2 directory or maliciously changes the contents of jars or other blackhat stuff to other running processes on the Jenkins server
          c) RM takes release artifacts, signs them and then puts them up for vote, completely unaware of what is actually in the package...

          In other words, doing releases on the Jenkins slaves is an absolutely terrible idea. Docker is not going to protect us here. (See also http://www.apache.org/dev/release.html#owned-controlled-hardware, which means if we're not breaking the letter of the law, we're definitely breaking the spirit...)

          Now that we've eliminated running this on jenkins for anything real, we're back to RMs running this on their own boxes. RM who are much more likely to be running things in parallel, even on the same branch....

          Show
          aw Allen Wittenauer added a comment - Regarding the maven cache, is this at all addressed by the release builds not being SNAPSHOT versions? All the precommit stuff should be SNAPSHOT, and I doubt there are multiple RMs building the same non-SNAPSHOT release version. I'll get to this in a second, but be aware there is still a process problem here: a) RM fires off a job on a shared machine to make a release b) Someone submits a patch that changes hadoop's or some other project's build such that it either mistakenly nukes the .m2 directory or maliciously changes the contents of jars or other blackhat stuff to other running processes on the Jenkins server c) RM takes release artifacts, signs them and then puts them up for vote, completely unaware of what is actually in the package... In other words, doing releases on the Jenkins slaves is an absolutely terrible idea. Docker is not going to protect us here. (See also http://www.apache.org/dev/release.html#owned-controlled-hardware , which means if we're not breaking the letter of the law, we're definitely breaking the spirit...) Now that we've eliminated running this on jenkins for anything real, we're back to RMs running this on their own boxes. RM who are much more likely to be running things in parallel, even on the same branch....
          Hide
          busbey Sean Busbey added a comment -

          the apache-release profile is usually defined in the apache parent pom (ex: the profile in org.apache:apache:17).

          It's usually set up to help make sure projects are generating valid release artifacts. It looks like Hadoop doesn't use the ASF parent pom though?

          Show
          busbey Sean Busbey added a comment - the apache-release profile is usually defined in the apache parent pom (ex: the profile in org.apache:apache:17 ). It's usually set up to help make sure projects are generating valid release artifacts. It looks like Hadoop doesn't use the ASF parent pom though?
          Hide
          aw Allen Wittenauer added a comment - - edited

          That was my assessment too.

          Hadoop's release process is ... not good. I've been (slowly) gearing up to make a run at a 3.x release for while now. Every corner I turn seems to be littered with more crud that needs to be cleaned up and manual processes that make no sense.

          (... and yes, the rewrite of the release notes builder 2-3 years ago is when I started on this journey... sigh.)

          Show
          aw Allen Wittenauer added a comment - - edited That was my assessment too. Hadoop's release process is ... not good. I've been (slowly) gearing up to make a run at a 3.x release for while now. Every corner I turn seems to be littered with more crud that needs to be cleaned up and manual processes that make no sense. (... and yes, the rewrite of the release notes builder 2-3 years ago is when I started on this journey... sigh.)
          Hide
          andrew.wang Andrew Wang added a comment -

          So, is the fix specifying an independent m2 dir? This means we end up downloading the world, but I've seen some good improvements by bumping the # of download threads and using Google's maven mirror.

          Show
          andrew.wang Andrew Wang added a comment - So, is the fix specifying an independent m2 dir? This means we end up downloading the world, but I've seen some good improvements by bumping the # of download threads and using Google's maven mirror.
          Hide
          aw Allen Wittenauer added a comment -

          So, is the fix specifying an independent m2 dir?

          With all the wrappings that need to go into making that work effectively, yes. That at least solves the maven cache problem.

          This means we end up downloading the world

          Let's not forget that this is for a release. Downloading the world is a good thing.

          BTW, don't forget that HADOOP-12893 still needs to be taken care of as well.

          Show
          aw Allen Wittenauer added a comment - So, is the fix specifying an independent m2 dir? With all the wrappings that need to go into making that work effectively, yes. That at least solves the maven cache problem. This means we end up downloading the world Let's not forget that this is for a release . Downloading the world is a good thing . BTW, don't forget that HADOOP-12893 still needs to be taken care of as well.
          Hide
          aw Allen Wittenauer added a comment - - edited

          I've been digging into our release artifacts today while testing my latest iteration of this and found some concerning things...

          From what I can tell, there are effectively 4 ways in which bits can appear in the final dist:

          • Use the dist-layout-stitching script (either in hadoop-dist/pom.xml in branch-2 or pulled out into dev-support in trunk)
          • Custom handling in hadoop-assemblies
          • Custom handling in that modules's pom.xml
          • Custom handling elsewhere that needs to be dug out (e.g., hadoop-mapreduce-project specifically handling the stuff in hadoop-mapreduce-client-nativetask)

          If a module is added that doesn't do any of these things, then they effectively disappear. For trunk, I've noticed two oddities:

          • hdfs-client source, test, and dependency jars are missing
          • hadoop-hdfs-native-client completely disappears

          There are likely more.

          Show
          aw Allen Wittenauer added a comment - - edited I've been digging into our release artifacts today while testing my latest iteration of this and found some concerning things... From what I can tell, there are effectively 4 ways in which bits can appear in the final dist: Use the dist-layout-stitching script (either in hadoop-dist/pom.xml in branch-2 or pulled out into dev-support in trunk) Custom handling in hadoop-assemblies Custom handling in that modules's pom.xml Custom handling elsewhere that needs to be dug out (e.g., hadoop-mapreduce-project specifically handling the stuff in hadoop-mapreduce-client-nativetask) If a module is added that doesn't do any of these things, then they effectively disappear. For trunk, I've noticed two oddities: hdfs-client source, test, and dependency jars are missing hadoop-hdfs-native-client completely disappears There are likely more.
          Hide
          aw Allen Wittenauer added a comment -

          -01:

          • stupid hacks to make hdfs-client* somewhat bundle better (dist-layout-stitching, pom component, cmake changes, etc)
          • maven cache support
          • better signing support
          • --asfrelease option now sets everything for actually making a release
          • quite a few bug fixes
          • took out the .gitignore changes
          • sets JAVA_HOME to the JVM version in the maven file when running Docker
          • when in --asfrelease mode, make sure the signing key is in the KEYS file
          Show
          aw Allen Wittenauer added a comment - -01: stupid hacks to make hdfs-client* somewhat bundle better (dist-layout-stitching, pom component, cmake changes, etc) maven cache support better signing support --asfrelease option now sets everything for actually making a release quite a few bug fixes took out the .gitignore changes sets JAVA_HOME to the JVM version in the maven file when running Docker when in --asfrelease mode, make sure the signing key is in the KEYS file
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          0 shelldocs 0m 4s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 46s Maven dependency ordering for branch
          +1 mvninstall 7m 46s trunk passed
          +1 compile 8m 23s trunk passed with JDK v1.8.0_74
          +1 compile 7m 43s trunk passed with JDK v1.7.0_95
          +1 mvnsite 0m 59s trunk passed
          +1 mvneclipse 0m 33s trunk passed
          +1 javadoc 0m 46s trunk passed with JDK v1.8.0_74
          +1 javadoc 0m 46s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 0m 52s the patch passed
          +1 compile 8m 30s the patch passed with JDK v1.8.0_74
          -1 cc 9m 38s root-jdk1.8.0_74 with JDK v1.8.0_74 generated 1 new + 10 unchanged - 1 fixed = 11 total (was 11)
          +1 cc 8m 30s the patch passed
          +1 javac 8m 30s the patch passed
          +1 compile 7m 44s the patch passed with JDK v1.7.0_95
          -1 cc 17m 22s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 20 unchanged - 1 fixed = 21 total (was 21)
          +1 cc 7m 44s the patch passed
          +1 javac 7m 44s the patch passed
          +1 mvnsite 0m 58s the patch passed
          +1 mvneclipse 0m 32s the patch passed
          +1 shellcheck 0m 10s There were no new shellcheck issues.
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 0m 45s the patch passed with JDK v1.8.0_74
          +1 javadoc 0m 44s the patch passed with JDK v1.7.0_95
          +1 unit 0m 8s hadoop-project-dist in the patch passed with JDK v1.8.0_74.
          +1 unit 1m 2s hadoop-hdfs-client in the patch passed with JDK v1.8.0_74.
          +1 unit 0m 38s hadoop-hdfs-native-client in the patch passed with JDK v1.8.0_74.
          +1 unit 0m 8s hadoop-project-dist in the patch passed with JDK v1.7.0_95.
          +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 37s hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          54m 16s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796636/HADOOP-12892.01.patch
          JIRA Issue HADOOP-12892
          Optional Tests asflicense shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall xml cc
          uname Linux 432b627f768d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 81d04ca
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          shellcheck v0.4.3
          cc root-jdk1.8.0_74: https://builds.apache.org/job/PreCommit-HADOOP-Build/9009/artifact/patchprocess/diff-compile-cc-root-jdk1.8.0_74.txt
          cc root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HADOOP-Build/9009/artifact/patchprocess/diff-compile-cc-root-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9009/testReport/
          modules C: hadoop-project-dist hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-native-client U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9009/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. 0 shelldocs 0m 4s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 46s Maven dependency ordering for branch +1 mvninstall 7m 46s trunk passed +1 compile 8m 23s trunk passed with JDK v1.8.0_74 +1 compile 7m 43s trunk passed with JDK v1.7.0_95 +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 33s trunk passed +1 javadoc 0m 46s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 46s trunk passed with JDK v1.7.0_95 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 0m 52s the patch passed +1 compile 8m 30s the patch passed with JDK v1.8.0_74 -1 cc 9m 38s root-jdk1.8.0_74 with JDK v1.8.0_74 generated 1 new + 10 unchanged - 1 fixed = 11 total (was 11) +1 cc 8m 30s the patch passed +1 javac 8m 30s the patch passed +1 compile 7m 44s the patch passed with JDK v1.7.0_95 -1 cc 17m 22s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 20 unchanged - 1 fixed = 21 total (was 21) +1 cc 7m 44s the patch passed +1 javac 7m 44s the patch passed +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 32s the patch passed +1 shellcheck 0m 10s There were no new shellcheck issues. +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 45s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 44s the patch passed with JDK v1.7.0_95 +1 unit 0m 8s hadoop-project-dist in the patch passed with JDK v1.8.0_74. +1 unit 1m 2s hadoop-hdfs-client in the patch passed with JDK v1.8.0_74. +1 unit 0m 38s hadoop-hdfs-native-client in the patch passed with JDK v1.8.0_74. +1 unit 0m 8s hadoop-project-dist in the patch passed with JDK v1.7.0_95. +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. +1 unit 0m 37s hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 54m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796636/HADOOP-12892.01.patch JIRA Issue HADOOP-12892 Optional Tests asflicense shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall xml cc uname Linux 432b627f768d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 81d04ca Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 shellcheck v0.4.3 cc root-jdk1.8.0_74: https://builds.apache.org/job/PreCommit-HADOOP-Build/9009/artifact/patchprocess/diff-compile-cc-root-jdk1.8.0_74.txt cc root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HADOOP-Build/9009/artifact/patchprocess/diff-compile-cc-root-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9009/testReport/ modules C: hadoop-project-dist hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-native-client U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9009/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the rev Allen, sorry for not getting to this sooner. Some review comments:

          • Typo "relase" in help text and the docker imgname
          • Shall we change "patchprocess" to "releaseprocess" or some such for LOGDIR?
          • -asfrelease sets NATIVE and SIGN, so the "-asfrelease requires --sign" validation will never trigger
          • Should we even support building on Darwin? I'd rather it just abort for unsupported platforms. Is windows supported for that matter?
          • The docker run command will pass along --dockercache, which then prints a spurious error msg:
          $ docker run -i -t --privileged -v /home/andrew/.gnupg:/home/andrew/.gnupg -v /home/andrew/dev/apache/trunk/patchprocess:/home/andrew/dev/apache/trunk/patchprocess -v /home/andrew/dev/apache/trunk/target/artifacts:/home/andrew/dev/apache/trunk/target/artifacts -v /home/andrew/dev/apache/trunk:/build/source -u andrew -w /build/source hadoop/createrelase:3.0.0-SNAPSHOT_norc /build/source/dev-support/bin/create-release --mvncache=/maven --asfrelease --dockercache --indocker
          ERROR: docker mode not enabled. Disabling dockercache.
          
          • DOCKERAN isn't set anywhere, remove this check?
          • MVNCACHE is ignored when --dockercache is specified and not --indocker right? That would be a good validation.
          • Nit: some extra newlines at the end of makearelease
          • When I did --asfrelease with --docker, it hung without output waiting for my GPG passphrase. I typed in my PW a few times, but it kept prompting, possibly because there wasn't a GPG agent running. Ended up Ctrl-D'ing it.
          • Related to previous, it'd be nice to be able to do --sign independent of the build. Means we should filter out ".asc" and ".mds" files in signartifacts in case it's invoked twice though. I moved signartifacts out of its if statement as a test, which worked.
          Show
          andrew.wang Andrew Wang added a comment - Thanks for the rev Allen, sorry for not getting to this sooner. Some review comments: Typo "relase" in help text and the docker imgname Shall we change "patchprocess" to "releaseprocess" or some such for LOGDIR? - asfrelease sets NATIVE and SIGN, so the " -asfrelease requires --sign" validation will never trigger Should we even support building on Darwin? I'd rather it just abort for unsupported platforms. Is windows supported for that matter? The docker run command will pass along --dockercache, which then prints a spurious error msg: $ docker run -i -t --privileged -v /home/andrew/.gnupg:/home/andrew/.gnupg -v /home/andrew/dev/apache/trunk/patchprocess:/home/andrew/dev/apache/trunk/patchprocess -v /home/andrew/dev/apache/trunk/target/artifacts:/home/andrew/dev/apache/trunk/target/artifacts -v /home/andrew/dev/apache/trunk:/build/source -u andrew -w /build/source hadoop/createrelase:3.0.0-SNAPSHOT_norc /build/source/dev-support/bin/create-release --mvncache=/maven --asfrelease --dockercache --indocker ERROR: docker mode not enabled. Disabling dockercache. DOCKERAN isn't set anywhere, remove this check? MVNCACHE is ignored when --dockercache is specified and not --indocker right? That would be a good validation. Nit: some extra newlines at the end of makearelease When I did --asfrelease with --docker, it hung without output waiting for my GPG passphrase. I typed in my PW a few times, but it kept prompting, possibly because there wasn't a GPG agent running. Ended up Ctrl-D'ing it. Related to previous, it'd be nice to be able to do --sign independent of the build. Means we should filter out ".asc" and ".mds" files in signartifacts in case it's invoked twice though. I moved signartifacts out of its if statement as a test, which worked.
          Hide
          aw Allen Wittenauer added a comment -

          Thanks for the review. Clearly some bugs. But some clarification/comments:

          Shall we change "patchprocess" to "releaseprocess" or some such for LOGDIR?

          patchprocess is in .gitignore and who knows where else and I didn't want to add another one to reduce the amount of pollution.

          --asfrelease sets NATIVE and SIGN, so the "--asfrelease requires --sign" validation will never trigger

          Actually, it can:

            if [[ ! -d "${HOME}/.gnupg" ]]; then
              hadoop_error "ERROR: No .gnupg dir. Disabling signing capability."
              SIGN=false
            fi
          ...
            if [[ "${ASFRELEASE}" = true ]]; then
              if [[ "${SIGN}" = false ]]; then
                hadoop_error "ERROR: --asfrelease requires --sign. Exiting."
                exit
              fi
          

          In other words, if there is no .gnupg directory, then override the user's request for signing support as a warning. But if an ASF release is requested, fail out.

          Should we even support building on Darwin? I'd rather it just abort for unsupported platforms. Is windows supported for that matter?

          There's no reason why someone can't use create-release to build non-Apache releases...

          Related to previous, it'd be nice to be able to do --sign independent of the build.

          IMO, but that's an outside of the Apache process and outside of create-release problem. I don't think I'd trust an RM that can't write a simple shell for loop. One of my goals here is to lock in stone the ASF release process since it's pretty clear that people screw it up all the time and/or do very bad things (like build them on Jenkins).

          Show
          aw Allen Wittenauer added a comment - Thanks for the review. Clearly some bugs. But some clarification/comments: Shall we change "patchprocess" to "releaseprocess" or some such for LOGDIR? patchprocess is in .gitignore and who knows where else and I didn't want to add another one to reduce the amount of pollution. --asfrelease sets NATIVE and SIGN, so the "--asfrelease requires --sign" validation will never trigger Actually, it can: if [[ ! -d "${HOME}/.gnupg" ]]; then hadoop_error "ERROR: No .gnupg dir. Disabling signing capability." SIGN= false fi ... if [[ "${ASFRELEASE}" = true ]]; then if [[ "${SIGN}" = false ]]; then hadoop_error "ERROR: --asfrelease requires --sign. Exiting." exit fi In other words, if there is no .gnupg directory, then override the user's request for signing support as a warning. But if an ASF release is requested, fail out. Should we even support building on Darwin? I'd rather it just abort for unsupported platforms. Is windows supported for that matter? There's no reason why someone can't use create-release to build non-Apache releases... Related to previous, it'd be nice to be able to do --sign independent of the build. IMO, but that's an outside of the Apache process and outside of create-release problem. I don't think I'd trust an RM that can't write a simple shell for loop. One of my goals here is to lock in stone the ASF release process since it's pretty clear that people screw it up all the time and/or do very bad things (like build them on Jenkins).
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks Allen, sounds reasonable overall. Looking forward to the rev.

          In other words, if there is no .gnupg directory, then override the user's request for signing support as a warning. But if an ASF release is requested, fail out.

          Missed that, gotcha. Maybe instead of ERROR we say WARNING when we aren't exiting, since ERRORs are typically fatal.

          There's no reason why someone can't use create-release to build non-Apache releases...

          Not sure who is using create-release to build non-ASF releases. Anyway, perhaps we only do OS enforcement when running with asfrelease? I think this fits with "lock in stone" as you say.

          Show
          andrew.wang Andrew Wang added a comment - Thanks Allen, sounds reasonable overall. Looking forward to the rev. In other words, if there is no .gnupg directory, then override the user's request for signing support as a warning. But if an ASF release is requested, fail out. Missed that, gotcha. Maybe instead of ERROR we say WARNING when we aren't exiting, since ERRORs are typically fatal. There's no reason why someone can't use create-release to build non-Apache releases... Not sure who is using create-release to build non-ASF releases. Anyway, perhaps we only do OS enforcement when running with asfrelease? I think this fits with "lock in stone" as you say.
          Hide
          aw Allen Wittenauer added a comment -

          -02:

          • a ton of fixes
          • need to add some bits to the dockerfile for gpg
          • reorganized dockerfile to make it easier to add stuff
          Show
          aw Allen Wittenauer added a comment - -02: a ton of fixes need to add some bits to the dockerfile for gpg reorganized dockerfile to make it easier to add stuff
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the testing environment has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9161/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9161/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 10m 10s Docker mode activated.
          0 shelldocs 0m 4s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 29s trunk passed
          +1 compile 5m 36s trunk passed with JDK v1.8.0_77
          +1 compile 6m 32s trunk passed with JDK v1.7.0_95
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 34s trunk passed
          +1 javadoc 0m 37s trunk passed with JDK v1.8.0_77
          +1 javadoc 0m 45s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 0m 48s the patch passed
          +1 compile 5m 32s the patch passed with JDK v1.8.0_77
          +1 cc 5m 32s the patch passed
          +1 javac 5m 32s the patch passed
          +1 compile 6m 31s the patch passed with JDK v1.7.0_95
          +1 cc 6m 31s the patch passed
          +1 javac 6m 31s the patch passed
          +1 mvnsite 0m 59s the patch passed
          +1 mvneclipse 0m 34s the patch passed
          +1 shellcheck 0m 9s There were no new shellcheck issues.
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 0m 37s the patch passed with JDK v1.8.0_77
          +1 javadoc 0m 45s the patch passed with JDK v1.7.0_95
          +1 unit 0m 8s hadoop-project-dist in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 31s hadoop-hdfs-native-client in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 9s hadoop-project-dist in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 56s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 31s hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          53m 20s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12800383/HADOOP-12892.02.patch
          JIRA Issue HADOOP-12892
          Optional Tests asflicense shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall xml cc
          uname Linux 97e55342401d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6fcde2e
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          shellcheck v0.4.3
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9161/testReport/
          modules C: hadoop-project-dist hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-native-client U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9161/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 10m 10s Docker mode activated. 0 shelldocs 0m 4s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 29s trunk passed +1 compile 5m 36s trunk passed with JDK v1.8.0_77 +1 compile 6m 32s trunk passed with JDK v1.7.0_95 +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 34s trunk passed +1 javadoc 0m 37s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 45s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 0m 48s the patch passed +1 compile 5m 32s the patch passed with JDK v1.8.0_77 +1 cc 5m 32s the patch passed +1 javac 5m 32s the patch passed +1 compile 6m 31s the patch passed with JDK v1.7.0_95 +1 cc 6m 31s the patch passed +1 javac 6m 31s the patch passed +1 mvnsite 0m 59s the patch passed +1 mvneclipse 0m 34s the patch passed +1 shellcheck 0m 9s There were no new shellcheck issues. +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 37s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 45s the patch passed with JDK v1.7.0_95 +1 unit 0m 8s hadoop-project-dist in the patch passed with JDK v1.8.0_77. +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77. +1 unit 0m 31s hadoop-hdfs-native-client in the patch passed with JDK v1.8.0_77. +1 unit 0m 9s hadoop-project-dist in the patch passed with JDK v1.7.0_95. +1 unit 0m 56s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. +1 unit 0m 31s hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 53m 20s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12800383/HADOOP-12892.02.patch JIRA Issue HADOOP-12892 Optional Tests asflicense shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall xml cc uname Linux 97e55342401d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6fcde2e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 shellcheck v0.4.3 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9161/testReport/ modules C: hadoop-project-dist hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-native-client U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9161/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          -03:

          • clean up some output
          • fix hdfs.h
          Show
          aw Allen Wittenauer added a comment - -03: clean up some output fix hdfs.h
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the testing environment has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9165/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9165/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          0 shelldocs 0m 4s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 30s trunk passed
          +1 compile 5m 37s trunk passed with JDK v1.8.0_77
          +1 compile 6m 33s trunk passed with JDK v1.7.0_95
          +1 mvnsite 1m 21s trunk passed
          +1 mvneclipse 0m 55s trunk passed
          +1 javadoc 0m 54s trunk passed with JDK v1.8.0_77
          +1 javadoc 1m 3s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 4s the patch passed
          +1 compile 5m 32s the patch passed with JDK v1.8.0_77
          +1 cc 5m 32s the patch passed
          +1 javac 5m 32s the patch passed
          +1 compile 6m 35s the patch passed with JDK v1.7.0_95
          +1 cc 6m 35s the patch passed
          +1 javac 6m 35s the patch passed
          +1 mvnsite 1m 22s the patch passed
          +1 mvneclipse 0m 54s the patch passed
          +1 shellcheck 0m 9s There were no new shellcheck issues.
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 0m 55s the patch passed with JDK v1.8.0_77
          +1 javadoc 1m 5s the patch passed with JDK v1.7.0_95
          +1 unit 0m 8s hadoop-project-dist in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 8s hadoop-assemblies in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 9s hadoop-assemblies in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 30s hadoop-hdfs-native-client in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 9s hadoop-project-dist in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 10s hadoop-assemblies in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 10s hadoop-assemblies in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 56s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 32s hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          47m 4s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12800427/HADOOP-12892.03.patch
          JIRA Issue HADOOP-12892
          Optional Tests asflicense shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall xml cc
          uname Linux 7f971d5b01f3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6fcde2e
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          shellcheck v0.4.3
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9165/testReport/
          modules C: hadoop-project-dist hadoop-assemblies hadoop-assemblies hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-native-client U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9165/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. 0 shelldocs 0m 4s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 30s trunk passed +1 compile 5m 37s trunk passed with JDK v1.8.0_77 +1 compile 6m 33s trunk passed with JDK v1.7.0_95 +1 mvnsite 1m 21s trunk passed +1 mvneclipse 0m 55s trunk passed +1 javadoc 0m 54s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 3s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 4s the patch passed +1 compile 5m 32s the patch passed with JDK v1.8.0_77 +1 cc 5m 32s the patch passed +1 javac 5m 32s the patch passed +1 compile 6m 35s the patch passed with JDK v1.7.0_95 +1 cc 6m 35s the patch passed +1 javac 6m 35s the patch passed +1 mvnsite 1m 22s the patch passed +1 mvneclipse 0m 54s the patch passed +1 shellcheck 0m 9s There were no new shellcheck issues. +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 55s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 5s the patch passed with JDK v1.7.0_95 +1 unit 0m 8s hadoop-project-dist in the patch passed with JDK v1.8.0_77. +1 unit 0m 8s hadoop-assemblies in the patch passed with JDK v1.8.0_77. +1 unit 0m 9s hadoop-assemblies in the patch passed with JDK v1.8.0_77. +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77. +1 unit 0m 30s hadoop-hdfs-native-client in the patch passed with JDK v1.8.0_77. +1 unit 0m 9s hadoop-project-dist in the patch passed with JDK v1.7.0_95. +1 unit 0m 10s hadoop-assemblies in the patch passed with JDK v1.7.0_95. +1 unit 0m 10s hadoop-assemblies in the patch passed with JDK v1.7.0_95. +1 unit 0m 56s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. +1 unit 0m 32s hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 47m 4s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12800427/HADOOP-12892.03.patch JIRA Issue HADOOP-12892 Optional Tests asflicense shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall xml cc uname Linux 7f971d5b01f3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6fcde2e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 shellcheck v0.4.3 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9165/testReport/ modules C: hadoop-project-dist hadoop-assemblies hadoop-assemblies hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-native-client U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9165/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the rev Allen, LGTM +1. Tested locally and seems to have worked. I see you chose not to do some of the option validations I recommended, but that's a matter of taste.

          Only nit is that there's an extra space in --asfrelease usage text. Minor enough that we can fix on at commit time? IDK if you have anything else feature-wise you want to add.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for the rev Allen, LGTM +1. Tested locally and seems to have worked. I see you chose not to do some of the option validations I recommended, but that's a matter of taste. Only nit is that there's an extra space in --asfrelease usage text. Minor enough that we can fix on at commit time? IDK if you have anything else feature-wise you want to add.
          Hide
          aw Allen Wittenauer added a comment -

          Thanks for the review.

          There are somethings I wanted to fix yet (e.g., fuse), but I can do those later.

          Committed to trunk.

          Show
          aw Allen Wittenauer added a comment - Thanks for the review. There are somethings I wanted to fix yet (e.g., fuse), but I can do those later. Committed to trunk.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9681 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9681/)
          HADOOP-12892. fix/rewrite create-release (aw) (aw: rev 7b1c37a13a55dc184f1b64439b9928b53e352ee7)

          • dev-support/create-release.sh
          • dev-support/bin/dist-layout-stitching
          • dev-support/bin/dist-copynativelibs
          • hadoop-hdfs-project/hadoop-hdfs-native-client/src/CMakeLists.txt
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • dev-support/docker/Dockerfile
          • hadoop-project-dist/pom.xml
          • hadoop-hdfs-project/hadoop-hdfs-client/pom.xml
          • hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
          • dev-support/bin/create-release
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9681 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9681/ ) HADOOP-12892 . fix/rewrite create-release (aw) (aw: rev 7b1c37a13a55dc184f1b64439b9928b53e352ee7) dev-support/create-release.sh dev-support/bin/dist-layout-stitching dev-support/bin/dist-copynativelibs hadoop-hdfs-project/hadoop-hdfs-native-client/src/CMakeLists.txt hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml dev-support/docker/Dockerfile hadoop-project-dist/pom.xml hadoop-hdfs-project/hadoop-hdfs-client/pom.xml hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml dev-support/bin/create-release
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          After this in..hadoop-hdfs-native-client compilation failing by throwing the following...

          F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
          [ERROR] -> [Help 1]
          org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: F:\GitCode\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\target\bin\RelWithDebInfo does not exist.
          around Ant part ...<copy todir="F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target/bin">... @ 14:98 in F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
                  at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
                  at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
                  at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
                  at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
                  at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
                  at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
                  at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
                  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:317)
                  at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:152)
                  at org.apache.maven.cli.MavenCli.execute(MavenCli.java:555)
                  at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214)
                  at org.apache.maven.cli.MavenCli.main(MavenCli.java:158)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  at java.lang.reflect.Method.invoke(Method.java:606)
                  at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
                  at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
                  at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
                  at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
          Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\target\bin\RelWithDebInfo does not exist.
          around Ant part ...<copy todir="F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target/bin">... @ 14:98 in F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
                  at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:355)
                  at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:106)
                  at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
                  ... 19 more
          

          IIUC, following fix should be fine,, didn't run in Linux environment....Andrew Wang and Allen Wittenauer if you people agree, will raise separate Jira and upload the patch..Thanks..

          @@ -148,7 +148,7 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd">
                               </exec>
                               <!-- Copy for inclusion in distribution. -->
                               <copy todir="${project.build.directory}/bin">
          -                      <fileset dir="${project.build.directory}/native/target/bin/RelWithDebInfo"/>
          +                      <fileset dir="${project.build.directory}/native/bin/RelWithDebInfo"/>
          
          
          Show
          brahmareddy Brahma Reddy Battula added a comment - After this in..hadoop-hdfs-native-client compilation failing by throwing the following... F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml [ERROR] -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: F:\GitCode\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\target\bin\RelWithDebInfo does not exist. around Ant part ...<copy todir="F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target/bin">... @ 14:98 in F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:317) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:152) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:555) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214) at org.apache.maven.cli.MavenCli.main(MavenCli.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\target\bin\RelWithDebInfo does not exist. around Ant part ...<copy todir="F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target/bin">... @ 14:98 in F:\Trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:355) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:106) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) ... 19 more IIUC, following fix should be fine,, didn't run in Linux environment.... Andrew Wang and Allen Wittenauer if you people agree, will raise separate Jira and upload the patch..Thanks.. @@ -148,7 +148,7 @@ http: //maven.apache.org/xsd/maven-4.0.0.xsd"> </exec> <!-- Copy for inclusion in distribution. --> <copy todir= "${project.build.directory}/bin" > - <fileset dir= "${project.build.directory}/ native /target/bin/RelWithDebInfo" /> + <fileset dir= "${project.build.directory}/ native /bin/RelWithDebInfo" />
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Commit date given as " Date: 05-03-2016 05:12:04 ". I think, it should be April-27th,2016.

          Revision: 7b1c37a13a55dc184f1b64439b9928b53e352ee7
          Author: Allen Wittenauer <aw@apache.org>
          Date: 05-03-2016 05:12:04
          Message:
          HADOOP-12892. fix/rewrite create-release (aw)
          
          Show
          brahmareddy Brahma Reddy Battula added a comment - Commit date given as " Date: 05-03-2016 05:12:04 ". I think, it should be April-27th,2016. Revision: 7b1c37a13a55dc184f1b64439b9928b53e352ee7 Author: Allen Wittenauer <aw@apache.org> Date: 05-03-2016 05:12:04 Message: HADOOP-12892. fix/rewrite create-release (aw)
          Hide
          busbey Sean Busbey added a comment -

          that is the author date, and presumably when Allen first committed work related to this fix. If you look at the full commit you'll see the commit date is in fact on the 27th of April:

          $ git log --format=fuller --grep HADOOP-12892
          commit 7b1c37a13a55dc184f1b64439b9928b53e352ee7
          Author:     Allen Wittenauer <aw@apache.org>
          AuthorDate: Fri Mar 4 15:42:04 2016 -0800
          Commit:     Allen Wittenauer <aw@apache.org>
          CommitDate: Wed Apr 27 08:38:22 2016 -0700
          
              HADOOP-12892. fix/rewrite create-release (aw)
          
          Show
          busbey Sean Busbey added a comment - that is the author date, and presumably when Allen first committed work related to this fix. If you look at the full commit you'll see the commit date is in fact on the 27th of April: $ git log --format=fuller --grep HADOOP-12892 commit 7b1c37a13a55dc184f1b64439b9928b53e352ee7 Author: Allen Wittenauer <aw@apache.org> AuthorDate: Fri Mar 4 15:42:04 2016 -0800 Commit: Allen Wittenauer <aw@apache.org> CommitDate: Wed Apr 27 08:38:22 2016 -0700 HADOOP-12892. fix/rewrite create-release (aw)
          Hide
          busbey Sean Busbey added a comment -

          And the canonical commit date you should look at is the message to the common-commits@hadoop mailing list:

          https://s.apache.org/hadoop-commit-7b1c37a13a55dc184f1b64439b9928b53e352ee7

          Show
          busbey Sean Busbey added a comment - And the canonical commit date you should look at is the message to the common-commits@hadoop mailing list: https://s.apache.org/hadoop-commit-7b1c37a13a55dc184f1b64439b9928b53e352ee7
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Sean Busbey thanks for calrification..

          I thought based on commit date, git log will be generated ( even in the Jira, Filed : Development ) which makes little confusion atleast to me..

          $ git log --grep HADOOP-12892
          commit 7b1c37a13a55dc184f1b64439b9928b53e352ee7
          Author: Allen Wittenauer <aw@apache.org>
          Date:   Fri Mar 4 15:42:04 2016 -0800
          
              HADOOP-12892. fix/rewrite create-release (aw)
          
          

          bytheway I came to know --format=fuller now..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Sean Busbey thanks for calrification.. I thought based on commit date, git log will be generated ( even in the Jira, Filed : Development ) which makes little confusion atleast to me.. $ git log --grep HADOOP-12892 commit 7b1c37a13a55dc184f1b64439b9928b53e352ee7 Author: Allen Wittenauer <aw@apache.org> Date: Fri Mar 4 15:42:04 2016 -0800 HADOOP-12892. fix/rewrite create-release (aw) bytheway I came to know --format=fuller now. .
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Raised HDFS-10353 to fix this compilation error in windows..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Raised HDFS-10353 to fix this compilation error in windows..
          Hide
          andrew.wang Andrew Wang added a comment -

          I'm reopening this since we need this in branch-2.8 to handle the new releasedocs generation. I made a quick cherry-pick attempt and the ISA-L stuff at least needs to be removed.

          Show
          andrew.wang Andrew Wang added a comment - I'm reopening this since we need this in branch-2.8 to handle the new releasedocs generation. I made a quick cherry-pick attempt and the ISA-L stuff at least needs to be removed.
          Hide
          cnauroth Chris Nauroth added a comment -

          I discovered this broke the Windows distro build, and I'm filing a patch on HADOOP-13149 to fix it.

          Show
          cnauroth Chris Nauroth added a comment - I discovered this broke the Windows distro build, and I'm filing a patch on HADOOP-13149 to fix it.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Andrew Wang, this is marked to be a incompatible change, do you know why it is marked to that? How can we back port an incompatible change to 2.8?

          Thanks,

          Show
          leftnoteasy Wangda Tan added a comment - Andrew Wang , this is marked to be a incompatible change, do you know why it is marked to that? How can we back port an incompatible change to 2.8? Thanks,
          Hide
          andrew.wang Andrew Wang added a comment -

          I think Allen was the one who flagged it as incompatible (so he can correct me if I'm wrong), but my understanding is that it's only incompatible for people who are building releases. This might affect Bigtop or distribution vendors, but in the end it should be producing the same src and bin tarballs.

          So I think it's safe to backport to 2.8, and probably doesn't need to be flagged as incompatible since it doesn't affect end users.

          Show
          andrew.wang Andrew Wang added a comment - I think Allen was the one who flagged it as incompatible (so he can correct me if I'm wrong), but my understanding is that it's only incompatible for people who are building releases. This might affect Bigtop or distribution vendors, but in the end it should be producing the same src and bin tarballs. So I think it's safe to backport to 2.8, and probably doesn't need to be flagged as incompatible since it doesn't affect end users.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Andrew Wang, sorry for my late response, I missed your comment above.

          I took a quick try to backport this patch to branch-2.8, it has a couple of conflicts. At least, it depends on HADOOP-12850, and HADOOP-12850 depends on more commits like HADOOP-10115, which again are marked as an incompatible change.

          Given a couple of incompatible dependencies of this patch, I felt risky to backport all of them to branch-2.8. Since create-release works in branch-2 before branch-2.8, do you know which JIRA introduced failures of create-release in branch-2? Can we revert problematic commits which leads to the failure?

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Andrew Wang , sorry for my late response, I missed your comment above. I took a quick try to backport this patch to branch-2.8, it has a couple of conflicts. At least, it depends on HADOOP-12850 , and HADOOP-12850 depends on more commits like HADOOP-10115 , which again are marked as an incompatible change. Given a couple of incompatible dependencies of this patch, I felt risky to backport all of them to branch-2.8. Since create-release works in branch-2 before branch-2.8, do you know which JIRA introduced failures of create-release in branch-2? Can we revert problematic commits which leads to the failure?
          Hide
          andrew.wang Andrew Wang added a comment -

          So one missing feature in the branch-2/branch-2.8 create-release script is that it doesn't use the new -Preleasedocs to build the CHANGES.txt file. I'm not sure how much work it is to add this support, but that's the alternative if we don't backport this JIRA.

          Could you provide some more context on what breaks if we don't include HADOOP-12850 and so on? It might be less work to tweak the backport to avoid these dependent JIRAs compared to adding Preleasedocs support to the old script.

          Show
          andrew.wang Andrew Wang added a comment - So one missing feature in the branch-2/branch-2.8 create-release script is that it doesn't use the new -Preleasedocs to build the CHANGES.txt file. I'm not sure how much work it is to add this support, but that's the alternative if we don't backport this JIRA. Could you provide some more context on what breaks if we don't include HADOOP-12850 and so on? It might be less work to tweak the backport to avoid these dependent JIRAs compared to adding Preleasedocs support to the old script.
          Hide
          aw Allen Wittenauer added a comment -

          Just a reminder that the old method of building on Jenkins violates ASF policy. Also:

          • I need to update the HowToRelease docs to use create-release
          • -Pdocs is missing from the build, just like it was in the original create-release.sh file.
          Show
          aw Allen Wittenauer added a comment - Just a reminder that the old method of building on Jenkins violates ASF policy. Also: I need to update the HowToRelease docs to use create-release -Pdocs is missing from the build, just like it was in the original create-release.sh file.
          Hide
          leftnoteasy Wangda Tan added a comment -

          If -PreleaseDocs is relatively easy to be fixed in branch-2/branch-2.8, we should look at it. Allen Wittenauer, could you add your points regarding to which JIRA we need to backport to branch-2 to support -Preleasedocs? I can try to backport them.

          Since I have very limited understanding of shell scripts, I don't really know what will be broken if we don't include HADOOP-12850.

          Is the create-release in branch-2 broken by HADOOP-11792?

          Show
          leftnoteasy Wangda Tan added a comment - If -PreleaseDocs is relatively easy to be fixed in branch-2/branch-2.8, we should look at it. Allen Wittenauer , could you add your points regarding to which JIRA we need to backport to branch-2 to support -Preleasedocs? I can try to backport them. Since I have very limited understanding of shell scripts, I don't really know what will be broken if we don't include HADOOP-12850 . Is the create-release in branch-2 broken by HADOOP-11792 ?
          Hide
          aw Allen Wittenauer added a comment -

          Sorry, branch-2 is not a priority of my volunteer time.

          Show
          aw Allen Wittenauer added a comment - Sorry, branch-2 is not a priority of my volunteer time.
          Hide
          aw Allen Wittenauer added a comment -

          I've updated HowToRelease based upon having dev-support/bin/create-release and fixing quite a few things that were missing (jdiff!) or wrong (people.apache.org!). Also note that I filed HADOOP-13245 to fix up a few more things in the source and add some functionality.

          Show
          aw Allen Wittenauer added a comment - I've updated HowToRelease based upon having dev-support/bin/create-release and fixing quite a few things that were missing (jdiff!) or wrong (people.apache.org!). Also note that I filed HADOOP-13245 to fix up a few more things in the source and add some functionality.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Attaching a patch for branch-2.

          Difference between 03 patch and 01.branch-2 patch

          • 03 patch fixes dist-layout-stitching but branch-2 doesn't have the file. The file was created by HADOOP-12850 but I didn't cherry-pick the commit because HADOOP-12850 is an incompatible change. I fixed hadoop-dist/pom.xml instead.
          • Removed ISA-L related code.
          Show
          ajisakaa Akira Ajisaka added a comment - Attaching a patch for branch-2. Difference between 03 patch and 01.branch-2 patch 03 patch fixes dist-layout-stitching but branch-2 doesn't have the file. The file was created by HADOOP-12850 but I didn't cherry-pick the commit because HADOOP-12850 is an incompatible change. I fixed hadoop-dist/pom.xml instead. Removed ISA-L related code.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the testing environment has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9692/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9692/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 10m 33s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 41s Maven dependency ordering for branch
          +1 mvninstall 6m 52s branch-2 passed
          +1 compile 5m 39s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 26s branch-2 passed with JDK v1.7.0_101
          +1 mvnsite 1m 35s branch-2 passed
          +1 mvneclipse 1m 8s branch-2 passed
          +1 javadoc 1m 6s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 15s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 15s the patch passed
          +1 compile 6m 57s the patch passed with JDK v1.8.0_91
          -1 cc 6m 57s root-jdk1.8.0_91 with JDK v1.8.0_91 generated 1 new + 5 unchanged - 1 fixed = 6 total (was 6)
          +1 javac 6m 57s the patch passed
          +1 compile 7m 20s the patch passed with JDK v1.7.0_101
          +1 cc 7m 20s the patch passed
          +1 javac 7m 20s the patch passed
          +1 mvnsite 1m 44s the patch passed
          +1 mvneclipse 1m 14s the patch passed
          -1 shellcheck 0m 9s The patch generated 5 new + 496 unchanged - 46 fixed = 501 total (was 542)
          -1 shelldocs 0m 9s The patch generated 34 new + 14 unchanged - 0 fixed = 48 total (was 14)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 javadoc 1m 16s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 23s the patch passed with JDK v1.7.0_101
          +1 unit 0m 11s hadoop-project-dist in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 13s hadoop-assemblies in the patch passed with JDK v1.7.0_101.
          +1 unit 1m 6s hadoop-hdfs-client in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 37s hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 14s hadoop-dist in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          64m 25s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:babe025
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808975/HADOOP-12892.01.branch-2.patch
          JIRA Issue HADOOP-12892
          Optional Tests asflicense shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall xml cc
          uname Linux 775c24d02976 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 67dbd4c
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          shellcheck v0.4.4
          cc https://builds.apache.org/job/PreCommit-HADOOP-Build/9692/artifact/patchprocess/diff-compile-cc-root-jdk1.8.0_91.txt
          shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/9692/artifact/patchprocess/diff-patch-shellcheck.txt
          shelldocs https://builds.apache.org/job/PreCommit-HADOOP-Build/9692/artifact/patchprocess/diff-patch-shelldocs.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9692/testReport/
          modules C: hadoop-project-dist hadoop-assemblies hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-native-client hadoop-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9692/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 10m 33s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 41s Maven dependency ordering for branch +1 mvninstall 6m 52s branch-2 passed +1 compile 5m 39s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 26s branch-2 passed with JDK v1.7.0_101 +1 mvnsite 1m 35s branch-2 passed +1 mvneclipse 1m 8s branch-2 passed +1 javadoc 1m 6s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 15s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 6m 57s the patch passed with JDK v1.8.0_91 -1 cc 6m 57s root-jdk1.8.0_91 with JDK v1.8.0_91 generated 1 new + 5 unchanged - 1 fixed = 6 total (was 6) +1 javac 6m 57s the patch passed +1 compile 7m 20s the patch passed with JDK v1.7.0_101 +1 cc 7m 20s the patch passed +1 javac 7m 20s the patch passed +1 mvnsite 1m 44s the patch passed +1 mvneclipse 1m 14s the patch passed -1 shellcheck 0m 9s The patch generated 5 new + 496 unchanged - 46 fixed = 501 total (was 542) -1 shelldocs 0m 9s The patch generated 34 new + 14 unchanged - 0 fixed = 48 total (was 14) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 javadoc 1m 16s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 23s the patch passed with JDK v1.7.0_101 +1 unit 0m 11s hadoop-project-dist in the patch passed with JDK v1.7.0_101. +1 unit 0m 13s hadoop-assemblies in the patch passed with JDK v1.7.0_101. +1 unit 1m 6s hadoop-hdfs-client in the patch passed with JDK v1.7.0_101. +1 unit 0m 37s hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101. +1 unit 0m 14s hadoop-dist in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 64m 25s Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808975/HADOOP-12892.01.branch-2.patch JIRA Issue HADOOP-12892 Optional Tests asflicense shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall xml cc uname Linux 775c24d02976 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 67dbd4c Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 shellcheck v0.4.4 cc https://builds.apache.org/job/PreCommit-HADOOP-Build/9692/artifact/patchprocess/diff-compile-cc-root-jdk1.8.0_91.txt shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/9692/artifact/patchprocess/diff-patch-shellcheck.txt shelldocs https://builds.apache.org/job/PreCommit-HADOOP-Build/9692/artifact/patchprocess/diff-patch-shelldocs.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9692/testReport/ modules C: hadoop-project-dist hadoop-assemblies hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-native-client hadoop-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9692/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Backport LGTM +1, thanks Akira. I'll assume you did appropriate local testing (running create-release with some different args). Will let you confirm and commit.

          Show
          andrew.wang Andrew Wang added a comment - Backport LGTM +1, thanks Akira. I'll assume you did appropriate local testing (running create-release with some different args). Will let you confirm and commit.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Akira Ajisaka working on this fix.

          I just tried to run the create-release locally (OSX) without any parameters. It fails by following errors:

          ****************************************************************************
                                      Staging the release
          ****************************************************************************
          
          
          $ mv /Users/wtan/sandbox/hadoop/target/staging/hadoop-project /Users/wtan/sandbox/hadoop/target/r2.9.0-SNAPSHOT/
          $ cd /Users/wtan/sandbox/hadoop/target/
          $ tar czpf hadoop-site-2.9.0-SNAPSHOT.tar.gz r2.9.0-SNAPSHOT/api r2.9.0-SNAPSHOT/css r2.9.0-SNAPSHOT/dependency-analysis.html r2.9.0-SNAPSHOT/hadoop-annotations r2.9.0-SNAPSHOT/hadoop-ant r2.9.0-SNAPSHOT/hadoop-archive-logs r2.9.0-SNAPSHOT/hadoop-archives r2.9.0-SNAPSHOT/hadoop-assemblies r2.9.0-SNAPSHOT/hadoop-auth r2.9.0-SNAPSHOT/hadoop-auth-examples r2.9.0-SNAPSHOT/hadoop-aws r2.9.0-SNAPSHOT/hadoop-azure r2.9.0-SNAPSHOT/hadoop-common-project r2.9.0-SNAPSHOT/hadoop-datajoin r2.9.0-SNAPSHOT/hadoop-dist r2.9.0-SNAPSHOT/hadoop-distcp r2.9.0-SNAPSHOT/hadoop-extras r2.9.0-SNAPSHOT/hadoop-gridmix r2.9.0-SNAPSHOT/hadoop-hdfs-bkjournal r2.9.0-SNAPSHOT/hadoop-hdfs-httpfs r2.9.0-SNAPSHOT/hadoop-hdfs-nfs r2.9.0-SNAPSHOT/hadoop-hdfs-project r2.9.0-SNAPSHOT/hadoop-kms r2.9.0-SNAPSHOT/hadoop-mapreduce r2.9.0-SNAPSHOT/hadoop-mapreduce-client r2.9.0-SNAPSHOT/hadoop-mapreduce-examples r2.9.0-SNAPSHOT/hadoop-maven-plugins r2.9.0-SNAPSHOT/hadoop-minicluster r2.9.0-SNAPSHOT/hadoop-minikdc r2.9.0-SNAPSHOT/hadoop-nfs r2.9.0-SNAPSHOT/hadoop-openstack r2.9.0-SNAPSHOT/hadoop-pipes r2.9.0-SNAPSHOT/hadoop-project-dist r2.9.0-SNAPSHOT/hadoop-rumen r2.9.0-SNAPSHOT/hadoop-sls r2.9.0-SNAPSHOT/hadoop-streaming r2.9.0-SNAPSHOT/hadoop-tools r2.9.0-SNAPSHOT/hadoop-yarn r2.9.0-SNAPSHOT/hadoop-yarn-project r2.9.0-SNAPSHOT/images r2.9.0-SNAPSHOT/index.html r2.9.0-SNAPSHOT/project-reports.html
          $ cd /Users/wtan/sandbox/hadoop
          $ cp -p /Users/wtan/sandbox/hadoop/hadoop-common-project/hadoop-common/src/site/markdown/release/2.9.0-SNAPSHOT/CHANGES*.md /Users/wtan/sandbox/hadoop/target/artifacts/CHANGES.md
          cp: /Users/wtan/sandbox/hadoop/hadoop-common-project/hadoop-common/src/site/markdown/release/2.9.0-SNAPSHOT/CHANGES*.md: No such file or directory
          
          Failed!
          

          Could you let me know which parameter you are using to do tests locally?

          Thanks,

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Akira Ajisaka working on this fix. I just tried to run the create-release locally (OSX) without any parameters. It fails by following errors: **************************************************************************** Staging the release **************************************************************************** $ mv /Users/wtan/sandbox/hadoop/target/staging/hadoop-project /Users/wtan/sandbox/hadoop/target/r2.9.0-SNAPSHOT/ $ cd /Users/wtan/sandbox/hadoop/target/ $ tar czpf hadoop-site-2.9.0-SNAPSHOT.tar.gz r2.9.0-SNAPSHOT/api r2.9.0-SNAPSHOT/css r2.9.0-SNAPSHOT/dependency-analysis.html r2.9.0-SNAPSHOT/hadoop-annotations r2.9.0-SNAPSHOT/hadoop-ant r2.9.0-SNAPSHOT/hadoop-archive-logs r2.9.0-SNAPSHOT/hadoop-archives r2.9.0-SNAPSHOT/hadoop-assemblies r2.9.0-SNAPSHOT/hadoop-auth r2.9.0-SNAPSHOT/hadoop-auth-examples r2.9.0-SNAPSHOT/hadoop-aws r2.9.0-SNAPSHOT/hadoop-azure r2.9.0-SNAPSHOT/hadoop-common-project r2.9.0-SNAPSHOT/hadoop-datajoin r2.9.0-SNAPSHOT/hadoop-dist r2.9.0-SNAPSHOT/hadoop-distcp r2.9.0-SNAPSHOT/hadoop-extras r2.9.0-SNAPSHOT/hadoop-gridmix r2.9.0-SNAPSHOT/hadoop-hdfs-bkjournal r2.9.0-SNAPSHOT/hadoop-hdfs-httpfs r2.9.0-SNAPSHOT/hadoop-hdfs-nfs r2.9.0-SNAPSHOT/hadoop-hdfs-project r2.9.0-SNAPSHOT/hadoop-kms r2.9.0-SNAPSHOT/hadoop-mapreduce r2.9.0-SNAPSHOT/hadoop-mapreduce-client r2.9.0-SNAPSHOT/hadoop-mapreduce-examples r2.9.0-SNAPSHOT/hadoop-maven-plugins r2.9.0-SNAPSHOT/hadoop-minicluster r2.9.0-SNAPSHOT/hadoop-minikdc r2.9.0-SNAPSHOT/hadoop-nfs r2.9.0-SNAPSHOT/hadoop-openstack r2.9.0-SNAPSHOT/hadoop-pipes r2.9.0-SNAPSHOT/hadoop-project-dist r2.9.0-SNAPSHOT/hadoop-rumen r2.9.0-SNAPSHOT/hadoop-sls r2.9.0-SNAPSHOT/hadoop-streaming r2.9.0-SNAPSHOT/hadoop-tools r2.9.0-SNAPSHOT/hadoop-yarn r2.9.0-SNAPSHOT/hadoop-yarn-project r2.9.0-SNAPSHOT/images r2.9.0-SNAPSHOT/index.html r2.9.0-SNAPSHOT/project-reports.html $ cd /Users/wtan/sandbox/hadoop $ cp -p /Users/wtan/sandbox/hadoop/hadoop-common-project/hadoop-common/src/site/markdown/release/2.9.0-SNAPSHOT/CHANGES*.md /Users/wtan/sandbox/hadoop/target/artifacts/CHANGES.md cp: /Users/wtan/sandbox/hadoop/hadoop-common-project/hadoop-common/src/site/markdown/release/2.9.0-SNAPSHOT/CHANGES*.md: No such file or directory Failed! Could you let me know which parameter you are using to do tests locally? Thanks,
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I hadn't tested locally, and I faced the same error. I'll look into the cause.

          Show
          ajisakaa Akira Ajisaka added a comment - I hadn't tested locally, and I faced the same error. I'll look into the cause.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I noticed

          • We need to set the version to 2.9.0 before running create-release in branch-2
          • 2.9.0/CHANGES*.md should be in ${basedir}/src/site/markdown/release but it is in patchprocess/. It seems that the workingDirectory does not work.
            hadoop-common-project/hadoop-common/pom.xml
                                    <executable>${basedir}/../../dev-support/bin/releasedocmaker</executable>
                                    <workingDirectory>src/site/markdown/release/</workingDirectory>
            
          Show
          ajisakaa Akira Ajisaka added a comment - I noticed We need to set the version to 2.9.0 before running create-release in branch-2 2.9.0/CHANGES*.md should be in ${basedir}/src/site/markdown/release but it is in patchprocess/ . It seems that the workingDirectory does not work. hadoop-common-project/hadoop-common/pom.xml <executable>${basedir}/../../dev-support/bin/releasedocmaker</executable> <workingDirectory>src/site/markdown/release/</workingDirectory>
          Hide
          ajisakaa Akira Ajisaka added a comment -

          yetus-wrapper changes the working directory to patchprocess/ and execute the script specified by the first argument, so the workingDirectory does not work.

          yetus-wrapper
          pushd "${HADOOP_PATCHPROCESS}" >/dev/null
          if [[ $? != 0 ]]; then
            yetus_error "ERROR: yetus-dl: Cannot pushd to ${HADOOP_PATCHPROCESS}"
            exit 1
          fi
          

          We need to copy 2.9.0/CHANGES*.md from patchprocess/.

          Show
          ajisakaa Akira Ajisaka added a comment - yetus-wrapper changes the working directory to patchprocess/ and execute the script specified by the first argument, so the workingDirectory does not work. yetus-wrapper pushd "${HADOOP_PATCHPROCESS}" >/dev/ null if [[ $? != 0 ]]; then yetus_error "ERROR: yetus-dl: Cannot pushd to ${HADOOP_PATCHPROCESS}" exit 1 fi We need to copy 2.9.0/CHANGES*.md from patchprocess/ .
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Updated the patch. Ran create-release successfully on my MBA.

          Show
          ajisakaa Akira Ajisaka added a comment - Updated the patch. Ran create-release successfully on my MBA.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the testing environment has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9702/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9702/console in case of problems.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          02.branch-2 patch:

          • Removed workingDirectory
          • Copy CHANGES*.md from patchprocess instead of workingDirectory
          • Create the destination directory before copying documents into the directory.
          Show
          ajisakaa Akira Ajisaka added a comment - 02.branch-2 patch: Removed workingDirectory Copy CHANGES*.md from patchprocess instead of workingDirectory Create the destination directory before copying documents into the directory.
          Hide
          ajisakaa Akira Ajisaka added a comment - - edited

          FYI: create-release --asfrelease option requires --docker option and the option requires HADOOP-13245.

          Show
          ajisakaa Akira Ajisaka added a comment - - edited FYI: create-release --asfrelease option requires --docker option and the option requires HADOOP-13245 .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 10m 34s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 38s Maven dependency ordering for branch
          +1 mvninstall 6m 27s branch-2 passed
          +1 compile 5m 50s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 23s branch-2 passed with JDK v1.7.0_101
          +1 mvnsite 2m 34s branch-2 passed
          +1 mvneclipse 1m 22s branch-2 passed
          +1 javadoc 2m 9s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 2m 22s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 1s the patch passed
          +1 compile 5m 46s the patch passed with JDK v1.8.0_91
          +1 cc 5m 46s the patch passed
          +1 javac 5m 46s the patch passed
          +1 compile 6m 21s the patch passed with JDK v1.7.0_101
          +1 cc 6m 21s the patch passed
          +1 javac 6m 21s the patch passed
          +1 mvnsite 2m 34s the patch passed
          +1 mvneclipse 1m 23s the patch passed
          -1 shellcheck 0m 8s The patch generated 5 new + 496 unchanged - 46 fixed = 501 total (was 542)
          -1 shelldocs 0m 8s The patch generated 34 new + 14 unchanged - 0 fixed = 48 total (was 14)
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 javadoc 2m 9s the patch passed with JDK v1.8.0_91
          +1 javadoc 2m 20s the patch passed with JDK v1.7.0_101
          +1 unit 0m 12s hadoop-project-dist in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 12s hadoop-assemblies in the patch passed with JDK v1.7.0_101.
          +1 unit 8m 22s hadoop-common in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 34s hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 12s hadoop-dist in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          84m 20s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:babe025
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809155/HADOOP-12892.02.branch-2.patch
          JIRA Issue HADOOP-12892
          Optional Tests asflicense shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall xml cc
          uname Linux 28e3b9bb0876 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 0b7388b
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          shellcheck v0.4.4
          shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/9702/artifact/patchprocess/diff-patch-shellcheck.txt
          shelldocs https://builds.apache.org/job/PreCommit-HADOOP-Build/9702/artifact/patchprocess/diff-patch-shelldocs.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9702/artifact/patchprocess/whitespace-eol.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9702/testReport/
          modules C: hadoop-project-dist hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-native-client hadoop-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9702/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 10m 34s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 38s Maven dependency ordering for branch +1 mvninstall 6m 27s branch-2 passed +1 compile 5m 50s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 23s branch-2 passed with JDK v1.7.0_101 +1 mvnsite 2m 34s branch-2 passed +1 mvneclipse 1m 22s branch-2 passed +1 javadoc 2m 9s branch-2 passed with JDK v1.8.0_91 +1 javadoc 2m 22s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 1s the patch passed +1 compile 5m 46s the patch passed with JDK v1.8.0_91 +1 cc 5m 46s the patch passed +1 javac 5m 46s the patch passed +1 compile 6m 21s the patch passed with JDK v1.7.0_101 +1 cc 6m 21s the patch passed +1 javac 6m 21s the patch passed +1 mvnsite 2m 34s the patch passed +1 mvneclipse 1m 23s the patch passed -1 shellcheck 0m 8s The patch generated 5 new + 496 unchanged - 46 fixed = 501 total (was 542) -1 shelldocs 0m 8s The patch generated 34 new + 14 unchanged - 0 fixed = 48 total (was 14) -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 2s The patch has no ill-formed XML file. +1 javadoc 2m 9s the patch passed with JDK v1.8.0_91 +1 javadoc 2m 20s the patch passed with JDK v1.7.0_101 +1 unit 0m 12s hadoop-project-dist in the patch passed with JDK v1.7.0_101. +1 unit 0m 12s hadoop-assemblies in the patch passed with JDK v1.7.0_101. +1 unit 8m 22s hadoop-common in the patch passed with JDK v1.7.0_101. +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_101. +1 unit 0m 34s hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101. +1 unit 0m 12s hadoop-dist in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 84m 20s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809155/HADOOP-12892.02.branch-2.patch JIRA Issue HADOOP-12892 Optional Tests asflicense shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall xml cc uname Linux 28e3b9bb0876 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 0b7388b Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 shellcheck v0.4.4 shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/9702/artifact/patchprocess/diff-patch-shellcheck.txt shelldocs https://builds.apache.org/job/PreCommit-HADOOP-Build/9702/artifact/patchprocess/diff-patch-shelldocs.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9702/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9702/testReport/ modules C: hadoop-project-dist hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-native-client hadoop-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9702/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          yetus-wrapper changes the working directory to patchprocess/ and execute the script specified by the first argument, so the workingDirectory does not work.

          It changes it back just prior to execution.

          Show
          aw Allen Wittenauer added a comment - yetus-wrapper changes the working directory to patchprocess/ and execute the script specified by the first argument, so the workingDirectory does not work. It changes it back just prior to execution.
          Hide
          aw Allen Wittenauer added a comment -

          (Although, now that releasedocmaker supports a location in yetus 0.3.0, we should probably change the pom.xml to use it.)

          Show
          aw Allen Wittenauer added a comment - (Although, now that releasedocmaker supports a location in yetus 0.3.0, we should probably change the pom.xml to use it.)
          Hide
          ajisakaa Akira Ajisaka added a comment -

          It changes it back just prior to execution.

          Thank you for the reply. This is true after HADOOP-12743, which it is not in branch-2. I'll backport this.

          Show
          ajisakaa Akira Ajisaka added a comment - It changes it back just prior to execution. Thank you for the reply. This is true after HADOOP-12743 , which it is not in branch-2. I'll backport this.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          03.branch-2 patch: After backporting HADOOP-12743, reverted the followings

          • Removed workingDirectory
          • Copy CHANGES*.md from patchprocess instead of workingDirectory
          Show
          ajisakaa Akira Ajisaka added a comment - 03.branch-2 patch: After backporting HADOOP-12743 , reverted the followings Removed workingDirectory Copy CHANGES*.md from patchprocess instead of workingDirectory
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the testing environment has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9706/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9706/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 39s Maven dependency ordering for branch
          +1 mvninstall 6m 42s branch-2 passed
          +1 compile 6m 51s branch-2 passed with JDK v1.8.0_91
          +1 compile 7m 15s branch-2 passed with JDK v1.7.0_101
          +1 mvnsite 1m 43s branch-2 passed
          +1 mvneclipse 1m 13s branch-2 passed
          +1 javadoc 1m 12s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 18s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 20s the patch passed
          +1 compile 6m 48s the patch passed with JDK v1.8.0_91
          +1 cc 6m 48s the patch passed
          +1 javac 6m 48s the patch passed
          +1 compile 7m 19s the patch passed with JDK v1.7.0_101
          +1 cc 7m 19s the patch passed
          +1 javac 7m 19s the patch passed
          +1 mvnsite 1m 37s the patch passed
          +1 mvneclipse 1m 15s the patch passed
          -1 shellcheck 0m 10s The patch generated 5 new + 496 unchanged - 46 fixed = 501 total (was 542)
          -1 shelldocs 0m 9s The patch generated 34 new + 14 unchanged - 0 fixed = 48 total (was 14)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          +1 javadoc 1m 13s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 22s the patch passed with JDK v1.7.0_101
          +1 unit 0m 11s hadoop-project-dist in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 12s hadoop-assemblies in the patch passed with JDK v1.7.0_101.
          +1 unit 1m 6s hadoop-hdfs-client in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 39s hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 13s hadoop-dist in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          56m 6s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:babe025
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809212/HADOOP-12892.03.branch-2.patch
          JIRA Issue HADOOP-12892
          Optional Tests asflicense shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall xml cc
          uname Linux 81988d5a1f80 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / a086fd9
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          shellcheck v0.4.4
          shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/9706/artifact/patchprocess/diff-patch-shellcheck.txt
          shelldocs https://builds.apache.org/job/PreCommit-HADOOP-Build/9706/artifact/patchprocess/diff-patch-shelldocs.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9706/testReport/
          modules C: hadoop-project-dist hadoop-assemblies hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-native-client hadoop-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9706/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 39s Maven dependency ordering for branch +1 mvninstall 6m 42s branch-2 passed +1 compile 6m 51s branch-2 passed with JDK v1.8.0_91 +1 compile 7m 15s branch-2 passed with JDK v1.7.0_101 +1 mvnsite 1m 43s branch-2 passed +1 mvneclipse 1m 13s branch-2 passed +1 javadoc 1m 12s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 18s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 20s the patch passed +1 compile 6m 48s the patch passed with JDK v1.8.0_91 +1 cc 6m 48s the patch passed +1 javac 6m 48s the patch passed +1 compile 7m 19s the patch passed with JDK v1.7.0_101 +1 cc 7m 19s the patch passed +1 javac 7m 19s the patch passed +1 mvnsite 1m 37s the patch passed +1 mvneclipse 1m 15s the patch passed -1 shellcheck 0m 10s The patch generated 5 new + 496 unchanged - 46 fixed = 501 total (was 542) -1 shelldocs 0m 9s The patch generated 34 new + 14 unchanged - 0 fixed = 48 total (was 14) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. +1 javadoc 1m 13s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 22s the patch passed with JDK v1.7.0_101 +1 unit 0m 11s hadoop-project-dist in the patch passed with JDK v1.7.0_101. +1 unit 0m 12s hadoop-assemblies in the patch passed with JDK v1.7.0_101. +1 unit 1m 6s hadoop-hdfs-client in the patch passed with JDK v1.7.0_101. +1 unit 0m 39s hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101. +1 unit 0m 13s hadoop-dist in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 56m 6s Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809212/HADOOP-12892.03.branch-2.patch JIRA Issue HADOOP-12892 Optional Tests asflicense shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall xml cc uname Linux 81988d5a1f80 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / a086fd9 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 shellcheck v0.4.4 shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/9706/artifact/patchprocess/diff-patch-shellcheck.txt shelldocs https://builds.apache.org/job/PreCommit-HADOOP-Build/9706/artifact/patchprocess/diff-patch-shelldocs.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9706/testReport/ modules C: hadoop-project-dist hadoop-assemblies hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-native-client hadoop-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9706/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Akira Ajisaka!

          I've tried to run create-release on my local box (without --docker) on branch-2.

          It succeeded and can generate artifacts. I've manually take a look at generated sites, and start a simple YARN cluster runs few distributed shell job, all succeeded.

          The patch has few conflicts while applying to branch-2.8. Could you update patch for branch-2.8? I can help with verification.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Akira Ajisaka ! I've tried to run create-release on my local box (without --docker) on branch-2. It succeeded and can generate artifacts. I've manually take a look at generated sites, and start a simple YARN cluster runs few distributed shell job, all succeeded. The patch has few conflicts while applying to branch-2.8. Could you update patch for branch-2.8? I can help with verification.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Rebased the patch for branch-2.8. I couldn't verified the patch because of the connection failure in releasedocmaker. I filed YETUS-419 and created a patch.

          Show
          ajisakaa Akira Ajisaka added a comment - Rebased the patch for branch-2.8. I couldn't verified the patch because of the connection failure in releasedocmaker. I filed YETUS-419 and created a patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the testing environment has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9720/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9720/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 31s Maven dependency ordering for branch
          +1 mvninstall 7m 39s branch-2.8 passed
          +1 compile 8m 0s branch-2.8 passed with JDK v1.8.0_91
          +1 compile 7m 49s branch-2.8 passed with JDK v1.7.0_101
          +1 mvnsite 1m 32s branch-2.8 passed
          +1 mvneclipse 1m 11s branch-2.8 passed
          +1 javadoc 1m 15s branch-2.8 passed with JDK v1.8.0_91
          +1 javadoc 1m 21s branch-2.8 passed with JDK v1.7.0_101
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 27s the patch passed
          +1 compile 7m 45s the patch passed with JDK v1.8.0_91
          +1 cc 7m 45s the patch passed
          +1 javac 7m 45s the patch passed
          +1 compile 8m 2s the patch passed with JDK v1.7.0_101
          +1 cc 8m 2s the patch passed
          +1 javac 8m 2s the patch passed
          +1 mvnsite 1m 39s the patch passed
          +1 mvneclipse 1m 11s the patch passed
          -1 shellcheck 0m 9s The patch generated 5 new + 496 unchanged - 46 fixed = 501 total (was 542)
          -1 shelldocs 0m 10s The patch generated 34 new + 14 unchanged - 0 fixed = 48 total (was 14)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          +1 javadoc 1m 12s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 18s the patch passed with JDK v1.7.0_101
          +1 unit 0m 12s hadoop-project-dist in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 12s hadoop-assemblies in the patch passed with JDK v1.7.0_101.
          +1 unit 1m 8s hadoop-hdfs-client in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 48s hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 15s hadoop-dist in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          60m 52s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c60792e
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809416/HADOOP-12892.01.branch-2.8.patch
          JIRA Issue HADOOP-12892
          Optional Tests asflicense shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall xml cc
          uname Linux 5b6b33c56f93 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.8 / faf9a9f
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          shellcheck v0.4.4
          shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/9720/artifact/patchprocess/diff-patch-shellcheck.txt
          shelldocs https://builds.apache.org/job/PreCommit-HADOOP-Build/9720/artifact/patchprocess/diff-patch-shelldocs.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9720/testReport/
          modules C: hadoop-project-dist hadoop-assemblies hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-native-client hadoop-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9720/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 31s Maven dependency ordering for branch +1 mvninstall 7m 39s branch-2.8 passed +1 compile 8m 0s branch-2.8 passed with JDK v1.8.0_91 +1 compile 7m 49s branch-2.8 passed with JDK v1.7.0_101 +1 mvnsite 1m 32s branch-2.8 passed +1 mvneclipse 1m 11s branch-2.8 passed +1 javadoc 1m 15s branch-2.8 passed with JDK v1.8.0_91 +1 javadoc 1m 21s branch-2.8 passed with JDK v1.7.0_101 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 7m 45s the patch passed with JDK v1.8.0_91 +1 cc 7m 45s the patch passed +1 javac 7m 45s the patch passed +1 compile 8m 2s the patch passed with JDK v1.7.0_101 +1 cc 8m 2s the patch passed +1 javac 8m 2s the patch passed +1 mvnsite 1m 39s the patch passed +1 mvneclipse 1m 11s the patch passed -1 shellcheck 0m 9s The patch generated 5 new + 496 unchanged - 46 fixed = 501 total (was 542) -1 shelldocs 0m 10s The patch generated 34 new + 14 unchanged - 0 fixed = 48 total (was 14) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. +1 javadoc 1m 12s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 18s the patch passed with JDK v1.7.0_101 +1 unit 0m 12s hadoop-project-dist in the patch passed with JDK v1.7.0_101. +1 unit 0m 12s hadoop-assemblies in the patch passed with JDK v1.7.0_101. +1 unit 1m 8s hadoop-hdfs-client in the patch passed with JDK v1.7.0_101. +1 unit 0m 48s hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101. +1 unit 0m 15s hadoop-dist in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 60m 52s Subsystem Report/Notes Docker Image:yetus/hadoop:c60792e JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809416/HADOOP-12892.01.branch-2.8.patch JIRA Issue HADOOP-12892 Optional Tests asflicense shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall xml cc uname Linux 5b6b33c56f93 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / faf9a9f Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 shellcheck v0.4.4 shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/9720/artifact/patchprocess/diff-patch-shellcheck.txt shelldocs https://builds.apache.org/job/PreCommit-HADOOP-Build/9720/artifact/patchprocess/diff-patch-shelldocs.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9720/testReport/ modules C: hadoop-project-dist hadoop-assemblies hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-native-client hadoop-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9720/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Akira Ajisaka, Thanks for rebasing the patch.

          I have tried this patch on branch-2.8, and artifacts can be built locally, I have also verified sites and launch YARN pseudo cluster using built binaries.

          For the fix itself, I will +1 since the problem is already fixed. Not sure about shell check and shell doc -1, should we treat it like an error or just a warning.

          I would like to get someone more familiar with this field to review and commit the patch.
          cc: Andrew Wang.

          Show
          leftnoteasy Wangda Tan added a comment - Akira Ajisaka , Thanks for rebasing the patch. I have tried this patch on branch-2.8, and artifacts can be built locally, I have also verified sites and launch YARN pseudo cluster using built binaries. For the fix itself, I will +1 since the problem is already fixed. Not sure about shell check and shell doc -1, should we treat it like an error or just a warning. I would like to get someone more familiar with this field to review and commit the patch. cc: Andrew Wang .
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi Andrew Wang, would you review the latest patches? (03.branch-2 and 01.branch-2.8)

          Show
          ajisakaa Akira Ajisaka added a comment - Hi Andrew Wang , would you review the latest patches? (03.branch-2 and 01.branch-2.8)
          Hide
          andrew.wang Andrew Wang added a comment -

          Took a quick look, LGTM thanks Akira, also thanks Wangda for validating.

          Show
          andrew.wang Andrew Wang added a comment - Took a quick look, LGTM thanks Akira, also thanks Wangda for validating.
          Hide
          andrew.wang Andrew Wang added a comment -

          Forgot to mention, I looked at the shellcheck/shell doc errors too, these exist in the trunk version too. I'm not sure why they weren't picked up by the original precommit run.

          Show
          andrew.wang Andrew Wang added a comment - Forgot to mention, I looked at the shellcheck/shell doc errors too, these exist in the trunk version too. I'm not sure why they weren't picked up by the original precommit run.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed to branch-2 and branch-2.8. Thanks Wangda Tan and Andrew Wang for review!

          Show
          ajisakaa Akira Ajisaka added a comment - Committed to branch-2 and branch-2.8. Thanks Wangda Tan and Andrew Wang for review!
          Hide
          ajisakaa Akira Ajisaka added a comment -

          After backporting this issue, we need to backport HDFS-10353, HADOOP-13149, and HADOOP-13245 as well. Let's do this.

          Show
          ajisakaa Akira Ajisaka added a comment - After backporting this issue, we need to backport HDFS-10353 , HADOOP-13149 , and HADOOP-13245 as well. Let's do this.

            People

            • Assignee:
              aw Allen Wittenauer
              Reporter:
              aw Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development