Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12662

The build should fail if a -Dbundle option fails

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      Per some discussion with Colin P. McCabe, it would be good to refine and make it consistent the behaviors in bundling native libraries when building dist package.
      For all native libraries to bundle, if the bundling option like -Dbundle.snappy is specified, then the lib option like -Dsnappy.lib will be checked and ensured to be there, but if not, it will then report error and fail the building explicitly.
      BUILDING.txt would also be updated to explicitly state this behavior.

      1. HADOOP-12662-v1.patch
        4 kB
        Kai Zheng
      2. HADOOP-12662-v2.patch
        4 kB
        Kai Zheng
      3. HADOOP-12662-v3.patch
        9 kB
        Kai Zheng
      4. HADOOP-12662-v4.patch
        9 kB
        Kai Zheng
      5. HADOOP-12662-v5.patch
        9 kB
        Kai Zheng

        Activity

        Hide
        drankye Kai Zheng added a comment -

        Uploaded a patch doing the work as described. Kindly give it a review. Thanks.

        Show
        drankye Kai Zheng added a comment - Uploaded a patch doing the work as described. Kindly give it a review. Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 58s trunk passed
        +1 compile 8m 41s trunk passed with JDK v1.8.0_66
        +1 compile 9m 25s trunk passed with JDK v1.7.0_91
        +1 mvnsite 10m 2s trunk passed
        +1 mvneclipse 0m 45s trunk passed
        +1 javadoc 5m 48s trunk passed with JDK v1.8.0_66
        +1 javadoc 9m 50s trunk passed with JDK v1.7.0_91
        +1 mvninstall 7m 54s the patch passed
        +1 compile 8m 42s the patch passed with JDK v1.8.0_66
        +1 javac 8m 42s the patch passed
        +1 compile 9m 26s the patch passed with JDK v1.7.0_91
        +1 javac 9m 26s the patch passed
        +1 mvnsite 9m 55s the patch passed
        +1 mvneclipse 0m 39s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 javadoc 5m 49s the patch passed with JDK v1.8.0_66
        +1 javadoc 9m 48s the patch passed with JDK v1.7.0_91
        -1 unit 11m 47s root in the patch failed with JDK v1.8.0_66.
        -1 unit 12m 21s root in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        130m 7s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
        JDK v1.7.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778775/HADOOP-12662-v1.patch
        JIRA Issue HADOOP-12662
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 6beff53684e3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 52ad912
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8285/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8285/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8285/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8285/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8285/testReport/
        modules C: . hadoop-project-dist U: .
        Max memory used 117MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8285/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 58s trunk passed +1 compile 8m 41s trunk passed with JDK v1.8.0_66 +1 compile 9m 25s trunk passed with JDK v1.7.0_91 +1 mvnsite 10m 2s trunk passed +1 mvneclipse 0m 45s trunk passed +1 javadoc 5m 48s trunk passed with JDK v1.8.0_66 +1 javadoc 9m 50s trunk passed with JDK v1.7.0_91 +1 mvninstall 7m 54s the patch passed +1 compile 8m 42s the patch passed with JDK v1.8.0_66 +1 javac 8m 42s the patch passed +1 compile 9m 26s the patch passed with JDK v1.7.0_91 +1 javac 9m 26s the patch passed +1 mvnsite 9m 55s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 javadoc 5m 49s the patch passed with JDK v1.8.0_66 +1 javadoc 9m 48s the patch passed with JDK v1.7.0_91 -1 unit 11m 47s root in the patch failed with JDK v1.8.0_66. -1 unit 12m 21s root in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 130m 7s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics JDK v1.7.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778775/HADOOP-12662-v1.patch JIRA Issue HADOOP-12662 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 6beff53684e3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 52ad912 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8285/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8285/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8285/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8285/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8285/testReport/ modules C: . hadoop-project-dist U: . Max memory used 117MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8285/console This message was automatically generated.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        s/faild/r/failed/

        Show
        stevel@apache.org Steve Loughran added a comment - s/faild/r/failed/
        Hide
        drankye Kai Zheng added a comment -

        Thanks Steve for the nice comment. It's done!

        Show
        drankye Kai Zheng added a comment - Thanks Steve for the nice comment. It's done!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 49s trunk passed
        +1 compile 8m 3s trunk passed with JDK v1.8.0_66
        +1 compile 8m 35s trunk passed with JDK v1.7.0_91
        +1 mvnsite 9m 32s trunk passed
        +1 mvneclipse 0m 50s trunk passed
        +1 javadoc 5m 12s trunk passed with JDK v1.8.0_66
        +1 javadoc 9m 1s trunk passed with JDK v1.7.0_91
        +1 mvninstall 7m 29s the patch passed
        +1 compile 7m 42s the patch passed with JDK v1.8.0_66
        +1 javac 7m 42s the patch passed
        +1 compile 8m 35s the patch passed with JDK v1.7.0_91
        +1 javac 8m 35s the patch passed
        +1 mvnsite 9m 5s the patch passed
        +1 mvneclipse 0m 36s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 javadoc 5m 15s the patch passed with JDK v1.8.0_66
        +1 javadoc 9m 5s the patch passed with JDK v1.7.0_91
        -1 unit 80m 20s root in the patch failed with JDK v1.8.0_66.
        -1 unit 81m 30s root in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 27s Patch does not generate ASF License warnings.
        260m 8s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestRecoverStripedFile



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778827/HADOOP-12662-v2.patch
        JIRA Issue HADOOP-12662
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 009bea2700fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2cb5aff
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8288/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8288/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8288/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8288/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8288/testReport/
        modules C: . hadoop-project-dist U: .
        Max memory used 117MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8288/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 49s trunk passed +1 compile 8m 3s trunk passed with JDK v1.8.0_66 +1 compile 8m 35s trunk passed with JDK v1.7.0_91 +1 mvnsite 9m 32s trunk passed +1 mvneclipse 0m 50s trunk passed +1 javadoc 5m 12s trunk passed with JDK v1.8.0_66 +1 javadoc 9m 1s trunk passed with JDK v1.7.0_91 +1 mvninstall 7m 29s the patch passed +1 compile 7m 42s the patch passed with JDK v1.8.0_66 +1 javac 7m 42s the patch passed +1 compile 8m 35s the patch passed with JDK v1.7.0_91 +1 javac 8m 35s the patch passed +1 mvnsite 9m 5s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 javadoc 5m 15s the patch passed with JDK v1.8.0_66 +1 javadoc 9m 5s the patch passed with JDK v1.7.0_91 -1 unit 80m 20s root in the patch failed with JDK v1.8.0_66. -1 unit 81m 30s root in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 260m 8s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestRecoverStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778827/HADOOP-12662-v2.patch JIRA Issue HADOOP-12662 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 009bea2700fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2cb5aff unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8288/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8288/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8288/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8288/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8288/testReport/ modules C: . hadoop-project-dist U: . Max memory used 117MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8288/console This message was automatically generated.
        Hide
        cmccabe Colin P. McCabe added a comment -

        I think this change makes sense. If bundling can't be done, and we requested it, the build should fail rather than silently creating something that isn't what we wanted (or even noisily... lets be honest, nobody is going to read the maven spew).

        One thing that is kind of annoying is that if the tar | untar pipeline fails, we continue on our merry way. For example, if the target directory runs out of disk space or the tar command is not executable, we'd silently create a partial build here.

        Unfortunately, checking the error statuses of commands in a shell pipeline is very difficult to do in a portable way.

        In bash, you could do something like this to capture the exit status of tar and untar:

        ( set -o pipefail; tar | untar ); STATUS=$?
        

        Do these install scripts require bash? If so, that would be a good way of making sure that we fail the build if tar or untar fails.

        Show
        cmccabe Colin P. McCabe added a comment - I think this change makes sense. If bundling can't be done, and we requested it, the build should fail rather than silently creating something that isn't what we wanted (or even noisily... lets be honest, nobody is going to read the maven spew). One thing that is kind of annoying is that if the tar | untar pipeline fails, we continue on our merry way. For example, if the target directory runs out of disk space or the tar command is not executable, we'd silently create a partial build here. Unfortunately, checking the error statuses of commands in a shell pipeline is very difficult to do in a portable way. In bash, you could do something like this to capture the exit status of tar and untar: ( set -o pipefail; tar | untar ); STATUS=$? Do these install scripts require bash? If so, that would be a good way of making sure that we fail the build if tar or untar fails.
        Hide
        drankye Kai Zheng added a comment -

        Do these install scripts require bash? If so, that would be a good way of making sure that we fail the build if tar or untar fails.

        Sounds good to use bash. I guess the dist package building is mainly used by Apache Bigtop? Andrew Purtell could you help comment on this? Thanks!

        Show
        drankye Kai Zheng added a comment - Do these install scripts require bash? If so, that would be a good way of making sure that we fail the build if tar or untar fails. Sounds good to use bash. I guess the dist package building is mainly used by Apache Bigtop? Andrew Purtell could you help comment on this? Thanks!
        Hide
        aw Allen Wittenauer added a comment -

        dist is used by anyone using the tar ball install. Also, we should pull all of this shell code out and put it into a script in dev-support if we're going to make it even more complicated than it already is. At least this way shellcheck can properly lint the code.

        Show
        aw Allen Wittenauer added a comment - dist is used by anyone using the tar ball install. Also, we should pull all of this shell code out and put it into a script in dev-support if we're going to make it even more complicated than it already is. At least this way shellcheck can properly lint the code.
        Hide
        drankye Kai Zheng added a comment -

        we should pull all of this shell code out and put it into a script in dev-support ...

        I would try it and see the effect. It would be much clean for the pom.xml. It looks bad with these script codes in my IDE.

        Show
        drankye Kai Zheng added a comment - we should pull all of this shell code out and put it into a script in dev-support ... I would try it and see the effect. It would be much clean for the pom.xml . It looks bad with these script codes in my IDE.
        Hide
        cmccabe Colin P. McCabe added a comment -

        Yeah, I think it's reasonable to split this script out into a separate file.

        Show
        cmccabe Colin P. McCabe added a comment - Yeah, I think it's reasonable to split this script out into a separate file.
        Hide
        drankye Kai Zheng added a comment -

        dist is used by anyone using the tar ball install.

        To clarify (please correct me if it's bad), the script only runs in tar ball building phase, while someone runs mvn packge with -Pdist -Dtar options. So the bash would be good to be required to be there in the building environment. It's not for end users. Even for hadoop deployment environment, I checked relevant daemon/client scripts, and they have already relied on bash.

        I think it's reasonable to split this script out into a separate file.

        With the script split out, it's a good chance to rewrite it (nice to use bash if agreed). For example, a function to bundle a native library would be nice to have, thus when adding a new native library, a simple line would be added to call it instead of having to duplicate a block of codes.

        Show
        drankye Kai Zheng added a comment - dist is used by anyone using the tar ball install. To clarify (please correct me if it's bad), the script only runs in tar ball building phase, while someone runs mvn packge with -Pdist -Dtar options. So the bash would be good to be required to be there in the building environment. It's not for end users. Even for hadoop deployment environment, I checked relevant daemon/client scripts, and they have already relied on bash. I think it's reasonable to split this script out into a separate file. With the script split out, it's a good chance to rewrite it (nice to use bash if agreed). For example, a function to bundle a native library would be nice to have, thus when adding a new native library, a simple line would be added to call it instead of having to duplicate a block of codes.
        Hide
        drankye Kai Zheng added a comment -

        In the root pom.xml, it was found:

        pom.xml:    <shell-executable>bash</shell-executable>
        

        And look at how it's used:

        hadoop-common-project/hadoop-common/pom.xml:                    <exec executable="${shell-executable}" failonerror="true" dir="${project.build.directory}/native">
        hadoop-common-project/hadoop-common/pom.xml:                    <exec executable="${shell-executable}" failonerror="true" dir="${project.build.directory}/native">
        hadoop-common-project/hadoop-kms/pom.xml:                    <exec executable="${shell-executable}" dir="${project.build.directory}"
        hadoop-common-project/hadoop-kms/pom.xml:                    <exec executable="${shell-executable}" dir="${project.build.directory}"
        hadoop-dist/pom.xml:                    <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true">
        hadoop-dist/pom.xml:                    <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true">
        hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:                    <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true">
        hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:                    <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true">
        hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml:                    <exec executable="${shell-executable}" failonerror="true" dir="${project.build.directory}/native/test">
        hadoop-mapreduce-project/pom.xml:                <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true">
        hadoop-mapreduce-project/pom.xml:                    <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true">
        hadoop-project-dist/pom.xml:                    <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true">
        hadoop-project-dist/pom.xml:                    <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true">
        hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml:                    <exec executable="${shell-executable}" failonerror="true" dir="${project.build.directory}/native">
        hadoop-yarn-project/pom.xml:                <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true">
        

        So it looks like we can safely use bash to rewrite the script.

        Show
        drankye Kai Zheng added a comment - In the root pom.xml , it was found: pom.xml: <shell-executable>bash</shell-executable> And look at how it's used: hadoop-common-project/hadoop-common/pom.xml: <exec executable="${shell-executable}" failonerror="true" dir="${project.build.directory}/native"> hadoop-common-project/hadoop-common/pom.xml: <exec executable="${shell-executable}" failonerror="true" dir="${project.build.directory}/native"> hadoop-common-project/hadoop-kms/pom.xml: <exec executable="${shell-executable}" dir="${project.build.directory}" hadoop-common-project/hadoop-kms/pom.xml: <exec executable="${shell-executable}" dir="${project.build.directory}" hadoop-dist/pom.xml: <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true"> hadoop-dist/pom.xml: <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true"> hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml: <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true"> hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml: <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true"> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml: <exec executable="${shell-executable}" failonerror="true" dir="${project.build.directory}/native/test"> hadoop-mapreduce-project/pom.xml: <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true"> hadoop-mapreduce-project/pom.xml: <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true"> hadoop-project-dist/pom.xml: <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true"> hadoop-project-dist/pom.xml: <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true"> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml: <exec executable="${shell-executable}" failonerror="true" dir="${project.build.directory}/native"> hadoop-yarn-project/pom.xml: <exec executable="${shell-executable}" dir="${project.build.directory}" failonerror="true"> So it looks like we can safely use bash to rewrite the script.
        Hide
        drankye Kai Zheng added a comment -

        As discussed it makes sense to separate the script codes out of hadoop-project-dist/pom.xml and I will proceed. We would also want to do the similar thing for hadoop-dist/pom.xml, if so maybe I would do it in separate issue?

        Show
        drankye Kai Zheng added a comment - As discussed it makes sense to separate the script codes out of hadoop-project-dist/pom.xml and I will proceed. We would also want to do the similar thing for hadoop-dist/pom.xml , if so maybe I would do it in separate issue?
        Hide
        cmccabe Colin P. McCabe added a comment -

        I agree that a separate script would be much "cleaner" because you'd be able to factor things out into subroutines, and run the various bash linters on it. I definitely don't have any objection to doing the factoring out in a different JIRA if that makes sense. I think bash is the right choice since we already use it in so many other places, and dealing with shell portability issues is just self-inflicted pain.

        Show
        cmccabe Colin P. McCabe added a comment - I agree that a separate script would be much "cleaner" because you'd be able to factor things out into subroutines, and run the various bash linters on it. I definitely don't have any objection to doing the factoring out in a different JIRA if that makes sense. I think bash is the right choice since we already use it in so many other places, and dealing with shell portability issues is just self-inflicted pain.
        Hide
        drankye Kai Zheng added a comment -

        Thanks Colin P. McCabe for the confirm. Wish you to have a wonderful vacation!

        I definitely don't have any objection to doing the factoring out in a different JIRA if that makes sense.

        To clarify, I'm going to do the refactoring thing for the native bundling script in hadoop-project-dist/pom.xml in this issue along with the tasks described here, and do the similar refactoring for hadoop-dist/pom.xml separately in another issue because it's a different thing.

        Show
        drankye Kai Zheng added a comment - Thanks Colin P. McCabe for the confirm. Wish you to have a wonderful vacation! I definitely don't have any objection to doing the factoring out in a different JIRA if that makes sense. To clarify, I'm going to do the refactoring thing for the native bundling script in hadoop-project-dist/pom.xml in this issue along with the tasks described here, and do the similar refactoring for hadoop-dist/pom.xml separately in another issue because it's a different thing.
        Hide
        drankye Kai Zheng added a comment -

        Updated the patch to:

        • Used bash and set -o pipefail to check error as Colin P. McCabe suggested;
        • Refactored the script codes adding a function to bundle a native library;
        • Fixed some minor issues related to bundling ISA-L found during the work.

        As to splitting the script codes out of pom.xml, it looks like there isn't obvious way to put the script file somewhere and reference it in the pom file, I suggest we consider the change separately. We may reconsider some basic approach for it according to somewhat best practice in Maven projects. After some experimenting, putting it in the dev-support folder looks like doesn't work elegantly.

        Show
        drankye Kai Zheng added a comment - Updated the patch to: Used bash and set -o pipefail to check error as Colin P. McCabe suggested; Refactored the script codes adding a function to bundle a native library; Fixed some minor issues related to bundling ISA-L found during the work. As to splitting the script codes out of pom.xml , it looks like there isn't obvious way to put the script file somewhere and reference it in the pom file, I suggest we consider the change separately. We may reconsider some basic approach for it according to somewhat best practice in Maven projects. After some experimenting, putting it in the dev-support folder looks like doesn't work elegantly.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 43s trunk passed
        +1 compile 10m 31s trunk passed with JDK v1.8.0_66
        +1 compile 9m 57s trunk passed with JDK v1.7.0_91
        +1 mvnsite 10m 41s trunk passed
        +1 mvneclipse 0m 41s trunk passed
        +1 javadoc 6m 34s trunk passed with JDK v1.8.0_66
        +1 javadoc 10m 31s trunk passed with JDK v1.7.0_91
        +1 mvninstall 8m 22s the patch passed
        +1 compile 9m 33s the patch passed with JDK v1.8.0_66
        +1 cc 9m 33s the patch passed
        +1 javac 9m 33s the patch passed
        +1 compile 10m 7s the patch passed with JDK v1.7.0_91
        +1 cc 10m 7s the patch passed
        +1 javac 10m 7s the patch passed
        +1 mvnsite 10m 48s the patch passed
        +1 mvneclipse 0m 41s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 javadoc 6m 57s the patch passed with JDK v1.8.0_66
        +1 javadoc 10m 49s the patch passed with JDK v1.7.0_91
        -1 unit 105m 6s root in the patch failed with JDK v1.8.0_66.
        -1 unit 87m 27s root in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 25s Patch does not generate ASF License warnings.
        309m 8s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDatanodeRegistration
          hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.server.namenode.TestNameNodeMXBean
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          hadoop.hdfs.TestRecoverStripedFile
          hadoop.hdfs.server.namenode.TestFsck
          hadoop.hdfs.qjournal.TestSecureNNWithQJM
          hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
          hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.shortcircuit.TestShortCircuitCache
          hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
        JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestFileCreation
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStream
          hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.shortcircuit.TestShortCircuitCache



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779629/HADOOP-12662-v3.patch
        JIRA Issue HADOOP-12662
        Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall xml
        uname Linux 5fe43193be49 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a0249da
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8315/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8315/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8315/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8315/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8315/testReport/
        modules C: . hadoop-common-project/hadoop-common hadoop-project-dist U: .
        Max memory used 115MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8315/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 43s trunk passed +1 compile 10m 31s trunk passed with JDK v1.8.0_66 +1 compile 9m 57s trunk passed with JDK v1.7.0_91 +1 mvnsite 10m 41s trunk passed +1 mvneclipse 0m 41s trunk passed +1 javadoc 6m 34s trunk passed with JDK v1.8.0_66 +1 javadoc 10m 31s trunk passed with JDK v1.7.0_91 +1 mvninstall 8m 22s the patch passed +1 compile 9m 33s the patch passed with JDK v1.8.0_66 +1 cc 9m 33s the patch passed +1 javac 9m 33s the patch passed +1 compile 10m 7s the patch passed with JDK v1.7.0_91 +1 cc 10m 7s the patch passed +1 javac 10m 7s the patch passed +1 mvnsite 10m 48s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 javadoc 6m 57s the patch passed with JDK v1.8.0_66 +1 javadoc 10m 49s the patch passed with JDK v1.7.0_91 -1 unit 105m 6s root in the patch failed with JDK v1.8.0_66. -1 unit 87m 27s root in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 309m 8s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDatanodeRegistration   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.TestNameNodeMXBean   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestFileCreation JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.shortcircuit.TestShortCircuitCache Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779629/HADOOP-12662-v3.patch JIRA Issue HADOOP-12662 Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall xml uname Linux 5fe43193be49 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a0249da Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8315/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8315/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8315/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8315/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8315/testReport/ modules C: . hadoop-common-project/hadoop-common hadoop-project-dist U: . Max memory used 115MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8315/console This message was automatically generated.
        Hide
        aw Allen Wittenauer added a comment -
        • If this is being switched to bash, then it should be using double brackets ([[) instead of single brackets ([) for tests. It will also make constructions like this:
        +                        if [ X"${bundle.snappy.in.bin}" = X"true" ] ; then
        

        significantly less painful since the X is no longer needed.

        • This is not now bash works.
        +                        set -o pipefail; $$TAR * | (cd $${TARGET_BIN_DIR}/; $$UNTAR)
        ...
        +                            set -o pipefail; $$TAR *snappy* | (cd $${TARGET_BIN_DIR}/; $$UNTAR)
        ...
        +                        set -o pipefail; $$TAR * | (cd $${TARGET_BIN_DIR}/; $$UNTAR)
        

        Once you set a bash option, it's set for the rest of the execution. So doing this in the middle is a debugging nightmare since the code will need to be traced to determine when pipefail was set. So you either need to not set it at all, set it at the beginning, or immediately unset it after you don't need it anymore.

        Show
        aw Allen Wittenauer added a comment - If this is being switched to bash, then it should be using double brackets ([[) instead of single brackets ([) for tests. It will also make constructions like this: + if [ X "${bundle.snappy.in.bin}" = X " true " ] ; then significantly less painful since the X is no longer needed. This is not now bash works. + set -o pipefail; $$TAR * | (cd $${TARGET_BIN_DIR}/; $$UNTAR) ... + set -o pipefail; $$TAR *snappy* | (cd $${TARGET_BIN_DIR}/; $$UNTAR) ... + set -o pipefail; $$TAR * | (cd $${TARGET_BIN_DIR}/; $$UNTAR) Once you set a bash option, it's set for the rest of the execution. So doing this in the middle is a debugging nightmare since the code will need to be traced to determine when pipefail was set. So you either need to not set it at all, set it at the beginning, or immediately unset it after you don't need it anymore.
        Hide
        drankye Kai Zheng added a comment -

        Thanks Allen Wittenauer for the great comments!

        significantly less painful since the X is no longer needed.

        I see. Maybe we could keep it as someone like me is used to the style?

        Once you set a bash option, ...,

        It sounds good to me to set it at the beginning thus it will be easier for the simple script.

        Show
        drankye Kai Zheng added a comment - Thanks Allen Wittenauer for the great comments! significantly less painful since the X is no longer needed. I see. Maybe we could keep it as someone like me is used to the style? Once you set a bash option, ..., It sounds good to me to set it at the beginning thus it will be easier for the simple script.
        Hide
        drankye Kai Zheng added a comment -

        A running instance bundling ISA-L library:

        [INFO] --- maven-antrun-plugin:1.7:run (pre-dist) @ hadoop-common ---
        [INFO] Executing tasks
        
        main:
             [exec] check_bundle_lib false snappy.lib snappy 
             [exec] Checking to bundle with:
             [exec] bundleOption=false, libOption=snappy.lib, libDir=, pattern=snappy
             [exec] check_bundle_lib false openssl.lib crypto 
             [exec] Checking to bundle with:
             [exec] bundleOption=false, libOption=openssl.lib, libDir=, pattern=crypto
             [exec] check_bundle_lib true isal.lib isa /usr/lib
             [exec] Checking to bundle with:
             [exec] bundleOption=true, libOption=isal.lib, libDir=/usr/lib, pattern=isa
        [INFO] Executed tasks
        
        Show
        drankye Kai Zheng added a comment - A running instance bundling ISA-L library: [INFO] --- maven-antrun-plugin:1.7:run (pre-dist) @ hadoop-common --- [INFO] Executing tasks main: [exec] check_bundle_lib false snappy.lib snappy [exec] Checking to bundle with: [exec] bundleOption=false, libOption=snappy.lib, libDir=, pattern=snappy [exec] check_bundle_lib false openssl.lib crypto [exec] Checking to bundle with: [exec] bundleOption=false, libOption=openssl.lib, libDir=, pattern=crypto [exec] check_bundle_lib true isal.lib isa /usr/lib [exec] Checking to bundle with: [exec] bundleOption=true, libOption=isal.lib, libDir=/usr/lib, pattern=isa [INFO] Executed tasks
        Hide
        drankye Kai Zheng added a comment -

        I met this when getting rid of the X, and wonder is there anything wrong in my setup, or we still need it?

        main:
             [exec] ./dist-copynativelibs.sh: line 16: unexpected argument `]]' to conditional binary operator
             [exec] ./dist-copynativelibs.sh: line 16: syntax error near `]]'
             [exec] ./dist-copynativelibs.sh: line 16: `                        if [[ "${libDir}" =  ]] || [[ ! -d ${libDir} ]]; then'
        
        Show
        drankye Kai Zheng added a comment - I met this when getting rid of the X , and wonder is there anything wrong in my setup, or we still need it? main: [exec] ./dist-copynativelibs.sh: line 16: unexpected argument `]]' to conditional binary operator [exec] ./dist-copynativelibs.sh: line 16: syntax error near `]]' [exec] ./dist-copynativelibs.sh: line 16: ` if [[ "${libDir}" = ]] || [[ ! -d ${libDir} ]]; then'
        Hide
        drankye Kai Zheng added a comment -

        A failure running instance bundling openssl failed:

        [INFO] --- maven-antrun-plugin:1.7:run (pre-dist) @ hadoop-common ---
        [INFO] Executing tasks
        
        main:
             [exec] check_bundle_lib true snappy.lib snappy /tmp/libsnappy
             [exec] Checking to bundle with:
             [exec] bundleOption=true, libOption=snappy.lib, libDir=/tmp/libsnappy, pattern=snappy
             [exec] check_bundle_lib true openssl.lib crypto /usr/lib64/openssl/engines
             [exec] Checking to bundle with:
             [exec] bundleOption=true, libOption=openssl.lib, libDir=/usr/lib64/openssl/engines, pattern=crypto
             [exec] Bundling library with openssl.lib failed tar: *crypto*: Cannot stat: No such file or directory
             [exec] tar: Exiting with failure status due to previous errors
             [exec] 
        [INFO] ------------------------------------------------------------------------
        
        Show
        drankye Kai Zheng added a comment - A failure running instance bundling openssl failed: [INFO] --- maven-antrun-plugin:1.7:run (pre-dist) @ hadoop-common --- [INFO] Executing tasks main: [exec] check_bundle_lib true snappy.lib snappy /tmp/libsnappy [exec] Checking to bundle with: [exec] bundleOption=true, libOption=snappy.lib, libDir=/tmp/libsnappy, pattern=snappy [exec] check_bundle_lib true openssl.lib crypto /usr/lib64/openssl/engines [exec] Checking to bundle with: [exec] bundleOption=true, libOption=openssl.lib, libDir=/usr/lib64/openssl/engines, pattern=crypto [exec] Bundling library with openssl.lib failed tar: *crypto*: Cannot stat: No such file or directory [exec] tar: Exiting with failure status due to previous errors [exec] [INFO] ------------------------------------------------------------------------
        Hide
        drankye Kai Zheng added a comment -

        A successful running instance bundling openssl, snappy and isal:

        [INFO] --- maven-antrun-plugin:1.7:run (pre-dist) @ hadoop-common ---
        [INFO] Executing tasks
        
        main:
             [exec] check_bundle_lib true snappy.lib snappy /tmp/libsnappy
             [exec] Checking to bundle with:
             [exec] bundleOption=true, libOption=snappy.lib, libDir=/tmp/libsnappy, pattern=snappy
             [exec] check_bundle_lib true openssl.lib crypto /usr/lib64
             [exec] Checking to bundle with:
             [exec] bundleOption=true, libOption=openssl.lib, libDir=/usr/lib64, pattern=crypto
             [exec] check_bundle_lib true isal.lib isa /usr/lib
             [exec] Checking to bundle with:
             [exec] bundleOption=true, libOption=isal.lib, libDir=/usr/lib, pattern=isa
        [INFO] Executed tasks
        [INFO] 
        

        And the bundle results:

        [root@zkdesk hadoop]# find hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/ -name *.so
        hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libcrypto.so
        hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libk5crypto.so
        hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libisal.so
        hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libhadoop.so
        hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libsnappy.so
        hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libnativetask.so
        
        Show
        drankye Kai Zheng added a comment - A successful running instance bundling openssl, snappy and isal: [INFO] --- maven-antrun-plugin:1.7:run (pre-dist) @ hadoop-common --- [INFO] Executing tasks main: [exec] check_bundle_lib true snappy.lib snappy /tmp/libsnappy [exec] Checking to bundle with: [exec] bundleOption=true, libOption=snappy.lib, libDir=/tmp/libsnappy, pattern=snappy [exec] check_bundle_lib true openssl.lib crypto /usr/lib64 [exec] Checking to bundle with: [exec] bundleOption=true, libOption=openssl.lib, libDir=/usr/lib64, pattern=crypto [exec] check_bundle_lib true isal.lib isa /usr/lib [exec] Checking to bundle with: [exec] bundleOption=true, libOption=isal.lib, libDir=/usr/lib, pattern=isa [INFO] Executed tasks [INFO] And the bundle results: [root@zkdesk hadoop]# find hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/ -name *.so hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libcrypto.so hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libk5crypto.so hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libisal.so hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libhadoop.so hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libsnappy.so hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libnativetask.so
        Hide
        drankye Kai Zheng added a comment -

        Updated the patch according to above discussion. Also tested as above.

        Show
        drankye Kai Zheng added a comment - Updated the patch according to above discussion. Also tested as above.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 52s trunk passed
        +1 compile 8m 39s trunk passed with JDK v1.8.0_66
        +1 compile 9m 24s trunk passed with JDK v1.7.0_91
        +1 mvnsite 10m 22s trunk passed
        +1 mvneclipse 0m 52s trunk passed
        +1 javadoc 7m 6s trunk passed with JDK v1.8.0_66
        +1 javadoc 10m 18s trunk passed with JDK v1.7.0_91
        +1 mvninstall 8m 18s the patch passed
        +1 compile 10m 14s the patch passed with JDK v1.8.0_66
        +1 cc 10m 14s the patch passed
        +1 javac 10m 14s the patch passed
        +1 compile 9m 54s the patch passed with JDK v1.7.0_91
        +1 cc 9m 54s the patch passed
        +1 javac 9m 54s the patch passed
        +1 mvnsite 10m 33s the patch passed
        +1 mvneclipse 0m 45s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 javadoc 7m 0s the patch passed with JDK v1.8.0_66
        +1 javadoc 10m 14s the patch passed with JDK v1.7.0_91
        -1 unit 11m 47s root in the patch failed with JDK v1.8.0_66.
        -1 unit 12m 19s root in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        137m 9s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
        JDK v1.7.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779759/HADOOP-12662-v4.patch
        JIRA Issue HADOOP-12662
        Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall xml
        uname Linux 30d4ca201396 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d0a22ba
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8321/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8321/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8321/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8321/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8321/testReport/
        modules C: . hadoop-common-project/hadoop-common hadoop-project-dist U: .
        Max memory used 117MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8321/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 52s trunk passed +1 compile 8m 39s trunk passed with JDK v1.8.0_66 +1 compile 9m 24s trunk passed with JDK v1.7.0_91 +1 mvnsite 10m 22s trunk passed +1 mvneclipse 0m 52s trunk passed +1 javadoc 7m 6s trunk passed with JDK v1.8.0_66 +1 javadoc 10m 18s trunk passed with JDK v1.7.0_91 +1 mvninstall 8m 18s the patch passed +1 compile 10m 14s the patch passed with JDK v1.8.0_66 +1 cc 10m 14s the patch passed +1 javac 10m 14s the patch passed +1 compile 9m 54s the patch passed with JDK v1.7.0_91 +1 cc 9m 54s the patch passed +1 javac 9m 54s the patch passed +1 mvnsite 10m 33s the patch passed +1 mvneclipse 0m 45s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 7m 0s the patch passed with JDK v1.8.0_66 +1 javadoc 10m 14s the patch passed with JDK v1.7.0_91 -1 unit 11m 47s root in the patch failed with JDK v1.8.0_66. -1 unit 12m 19s root in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 137m 9s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics JDK v1.7.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779759/HADOOP-12662-v4.patch JIRA Issue HADOOP-12662 Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall xml uname Linux 30d4ca201396 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d0a22ba Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8321/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8321/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8321/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8321/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8321/testReport/ modules C: . hadoop-common-project/hadoop-common hadoop-project-dist U: . Max memory used 117MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8321/console This message was automatically generated.
        Hide
        cmccabe Colin P. McCabe added a comment -

        Thanks for working on this, Kai Zheng.

        328                      # Bundle a native library if requested. Exit 1 in case error happens.
        329	                      # Usage: check_bundle_lib bundleOption libOption libPattern libDir
        330	                      function check_bundle_lib() {
        

        It seems like this should be named bundle_native_lib or something like that, rather than check_bundle_lib. It does more than just "checking," right? It actually performs the bundling.

        343                        if [[ X"${libDir}" = X ]] || [[ ! -d ${libDir} ]]; then
        344	                          echo "The required option $libOption isn't given or invalid. Bundling the lib failed"
        345	                          exit 1
        346	                        fi
        

        As Allen Wittenauer pointed out, the 'X' construction isn't needed in bash. Suggest using this instead:

        if [[ -z ${libDir} ]] || [[ ! -d ${libDir} ]]; then
        ...
        
        404                        if [[ X"${bundle.openssl.in.bin}" = X"true" ]]; then
        405	                          if [[ X"${bundle.openssl}" = X"true" ]]; then
        

        try using this instead (note the double equals sign):

        404                        if [[ "${bundle.openssl.in.bin}" == "true" ]]; then
        405	                          if [[ "${bundle.openssl}" == "true" ]]; then
        

        +1 once those are addressed. Perhaps we should open a follow-on jira to split this out into a separate bash file, which would be a bit cleaner.

        Show
        cmccabe Colin P. McCabe added a comment - Thanks for working on this, Kai Zheng . 328 # Bundle a native library if requested. Exit 1 in case error happens. 329 # Usage: check_bundle_lib bundleOption libOption libPattern libDir 330 function check_bundle_lib() { It seems like this should be named bundle_native_lib or something like that, rather than check_bundle_lib . It does more than just "checking," right? It actually performs the bundling. 343 if [[ X "${libDir}" = X ]] || [[ ! -d ${libDir} ]]; then 344 echo "The required option $libOption isn't given or invalid. Bundling the lib failed" 345 exit 1 346 fi As Allen Wittenauer pointed out, the 'X' construction isn't needed in bash. Suggest using this instead: if [[ -z ${libDir} ]] || [[ ! -d ${libDir} ]]; then ... 404 if [[ X "${bundle.openssl.in.bin}" = X " true " ]]; then 405 if [[ X "${bundle.openssl}" = X " true " ]]; then try using this instead (note the double equals sign): 404 if [[ "${bundle.openssl.in.bin}" == " true " ]]; then 405 if [[ "${bundle.openssl}" == " true " ]]; then +1 once those are addressed. Perhaps we should open a follow-on jira to split this out into a separate bash file, which would be a bit cleaner.
        Hide
        drankye Kai Zheng added a comment -

        Thanks Colin P. McCabe for the review and nice comments!

        It seems like this should be named bundle_native_lib or something like that, rather than check_bundle_lib

        Sounds much clearer to use bundle_native_lib as the checking is just the essential part of it.

        the 'X' construction isn't needed in bash. Suggest using this instead

        Ah, see. Thanks for the reminding and sample codes. I just got what Allen Wittenauer meant.

        try using this instead (note the double equals sign)

        Sure. Glad I can be refreshed in my shell aspect.

        Will update the patch accordingly.

        Show
        drankye Kai Zheng added a comment - Thanks Colin P. McCabe for the review and nice comments! It seems like this should be named bundle_native_lib or something like that, rather than check_bundle_lib Sounds much clearer to use bundle_native_lib as the checking is just the essential part of it. the 'X' construction isn't needed in bash. Suggest using this instead Ah, see. Thanks for the reminding and sample codes. I just got what Allen Wittenauer meant. try using this instead (note the double equals sign) Sure. Glad I can be refreshed in my shell aspect. Will update the patch accordingly.
        Hide
        drankye Kai Zheng added a comment -

        Perhaps we should open a follow-on jira to split this out into a separate bash file, which would be a bit cleaner.

        Yeah. I didn't make the change in the patch because after some trying it was found doing so involves change in bigger scope and better to be further discussed separately. I will create the issue to follow on this.

        Show
        drankye Kai Zheng added a comment - Perhaps we should open a follow-on jira to split this out into a separate bash file, which would be a bit cleaner. Yeah. I didn't make the change in the patch because after some trying it was found doing so involves change in bigger scope and better to be further discussed separately. I will create the issue to follow on this.
        Hide
        drankye Kai Zheng added a comment -

        Updated the patch according to above discussions.

        By the way, HADOOP-12703 was opened to follow on the script splitting issue.

        Show
        drankye Kai Zheng added a comment - Updated the patch according to above discussions. By the way, HADOOP-12703 was opened to follow on the script splitting issue.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 1s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 48s trunk passed
        +1 compile 7m 55s trunk passed with JDK v1.8.0_66
        +1 compile 8m 44s trunk passed with JDK v1.7.0_91
        +1 mvnsite 9m 48s trunk passed
        +1 mvneclipse 0m 54s trunk passed
        +1 javadoc 5m 32s trunk passed with JDK v1.8.0_66
        +1 javadoc 9m 24s trunk passed with JDK v1.7.0_91
        +1 mvninstall 7m 34s the patch passed
        +1 compile 7m 55s the patch passed with JDK v1.8.0_66
        +1 cc 7m 55s the patch passed
        +1 javac 7m 55s the patch passed
        +1 compile 8m 54s the patch passed with JDK v1.7.0_91
        +1 cc 8m 54s the patch passed
        +1 javac 8m 54s the patch passed
        +1 mvnsite 9m 30s the patch passed
        +1 mvneclipse 0m 41s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 javadoc 5m 38s the patch passed with JDK v1.8.0_66
        +1 javadoc 9m 18s the patch passed with JDK v1.7.0_91
        -1 unit 106m 49s root in the patch failed with JDK v1.8.0_66.
        -1 unit 110m 50s root in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 29s Patch does not generate ASF License warnings.
        318m 54s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.web.TestWebHDFS
          hadoop.hdfs.server.namenode.TestStartup
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestStartup
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781968/HADOOP-12662-v5.patch
        JIRA Issue HADOOP-12662
        Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall xml
        uname Linux 17fb55bfdc39 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / fbb5868
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8396/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8396/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8396/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8396/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8396/testReport/
        modules C: . hadoop-common-project/hadoop-common hadoop-project-dist U: .
        Max memory used 115MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8396/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 1s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 48s trunk passed +1 compile 7m 55s trunk passed with JDK v1.8.0_66 +1 compile 8m 44s trunk passed with JDK v1.7.0_91 +1 mvnsite 9m 48s trunk passed +1 mvneclipse 0m 54s trunk passed +1 javadoc 5m 32s trunk passed with JDK v1.8.0_66 +1 javadoc 9m 24s trunk passed with JDK v1.7.0_91 +1 mvninstall 7m 34s the patch passed +1 compile 7m 55s the patch passed with JDK v1.8.0_66 +1 cc 7m 55s the patch passed +1 javac 7m 55s the patch passed +1 compile 8m 54s the patch passed with JDK v1.7.0_91 +1 cc 8m 54s the patch passed +1 javac 8m 54s the patch passed +1 mvnsite 9m 30s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 javadoc 5m 38s the patch passed with JDK v1.8.0_66 +1 javadoc 9m 18s the patch passed with JDK v1.7.0_91 -1 unit 106m 49s root in the patch failed with JDK v1.8.0_66. -1 unit 110m 50s root in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 29s Patch does not generate ASF License warnings. 318m 54s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.web.TestWebHDFS   hadoop.hdfs.server.namenode.TestStartup JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestStartup   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781968/HADOOP-12662-v5.patch JIRA Issue HADOOP-12662 Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall xml uname Linux 17fb55bfdc39 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fbb5868 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8396/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8396/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8396/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8396/artifact/patchprocess/patch-unit-root-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8396/testReport/ modules C: . hadoop-common-project/hadoop-common hadoop-project-dist U: . Max memory used 115MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8396/console This message was automatically generated.
        Hide
        drankye Kai Zheng added a comment -

        As discussed in YETUS-222, currently there are no Jenkins building that triggers and tests the dist package building. For previous patch I made a thorough tests as mentioned above. For the latest update, I did a test and it passed locally.

        Show
        drankye Kai Zheng added a comment - As discussed in YETUS-222 , currently there are no Jenkins building that triggers and tests the dist package building. For previous patch I made a thorough tests as mentioned above. For the latest update, I did a test and it passed locally.
        Hide
        vinayrpet Vinayakumar B added a comment -

        Latest patch looks cleaner
        +1.

        Show
        vinayrpet Vinayakumar B added a comment - Latest patch looks cleaner +1.
        Hide
        cmccabe Colin P. McCabe added a comment -

        +1. I'll commit this tomorrow if nobody has any more comments.

        Thanks, Kai Zheng.

        Show
        cmccabe Colin P. McCabe added a comment - +1. I'll commit this tomorrow if nobody has any more comments. Thanks, Kai Zheng .
        Hide
        cmccabe Colin P. McCabe added a comment -

        Committed to 2.9. Thanks, Kai Zheng.

        Show
        cmccabe Colin P. McCabe added a comment - Committed to 2.9. Thanks, Kai Zheng .
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9125 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9125/)
        HADOOP-12662. The build should fail if a -Dbundle option fails (Kai (cmccabe: rev 2a303865391657648281270114bccc32fd4c051a)

        • BUILDING.txt
        • hadoop-project-dist/pom.xml
        • hadoop-common-project/hadoop-common/src/CMakeLists.txt
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9125 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9125/ ) HADOOP-12662 . The build should fail if a -Dbundle option fails (Kai (cmccabe: rev 2a303865391657648281270114bccc32fd4c051a) BUILDING.txt hadoop-project-dist/pom.xml hadoop-common-project/hadoop-common/src/CMakeLists.txt hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        drankye Kai Zheng added a comment -

        Thanks Colin P. McCabe, Steve Loughran, Allen Wittenauer and Vinayakumar B for the review, comments and commit!

        Show
        drankye Kai Zheng added a comment - Thanks Colin P. McCabe , Steve Loughran , Allen Wittenauer and Vinayakumar B for the review, comments and commit!

          People

          • Assignee:
            drankye Kai Zheng
            Reporter:
            drankye Kai Zheng
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development