Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: build
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      <!-- markdown -->
      * --build-native=false should work now
      * --branch option lets one specify a branch to test against on the command line
      * On certain Jenkins machines, the artifact directory sometimes gets deleted from outside the test-patch script. There is now some code to try to detect, alert, and quick exit if that happens.
      * Various semi-critical output and bug fixes
      Show
      <!-- markdown --> * --build-native=false should work now * --branch option lets one specify a branch to test against on the command line * On certain Jenkins machines, the artifact directory sometimes gets deleted from outside the test-patch script. There is now some code to try to detect, alert, and quick exit if that happens. * Various semi-critical output and bug fixes

      Description

      if you specify "--build-native=false" like

      ./dev-support/test-patch.sh  --build-native=false ~/workspaces/patches/hdfs-8211.001.patch 
      

      mvn fails with invalid lifecycle error.

      Here are the steps to repro :

      1) run any patch with --buid-native=false option

      2) Open up /tmp/hadoop-test-patch/<tmp-patch>/patchJavacWarnings.txt to see the failure reason.

      1. HADOOP-11861-00.patch
        8 kB
        Allen Wittenauer
      2. HADOOP-11861-01.patch
        10 kB
        Allen Wittenauer
      3. HADOOP-11861-02.patch
        11 kB
        Allen Wittenauer
      4. HADOOP-11861-04.patch
        16 kB
        Allen Wittenauer

        Issue Links

          Activity

          Hide
          anu Anu Engineer added a comment -

          Allen Wittenauer would you like to take a look at this to see if this is related to the new changes to build ?

          Show
          anu Anu Engineer added a comment - Allen Wittenauer would you like to take a look at this to see if this is related to the new changes to build ?
          Hide
          cnauroth Chris Nauroth added a comment -

          On this line, NATIVE_PROFILE would be undefined.

            echo_and_redirect "${PATCH_DIR}/patchJavacWarnings.txt" "${MVN}" clean test -DskipTests -D${PROJECT_NAME}PatchProcess "${NATIVE_PROFILE}" -Ptest-patch
          

          It seems like we end up passing an extra empty string argument to the mvn call, and mvn interprets that as an extra goal to run, which of course doesn't exist.

          /opt/local/share/java/maven3/bin/mvn clean test -DskipTests -DhadoopPatchProcess  -Ptest-patch > /tmp/hadoop-test-patch/2621/patchJavacWarnings.txt 2>&1
          
          Show
          cnauroth Chris Nauroth added a comment - On this line, NATIVE_PROFILE would be undefined. echo_and_redirect "${PATCH_DIR}/patchJavacWarnings.txt" "${MVN}" clean test -DskipTests -D${PROJECT_NAME}PatchProcess "${NATIVE_PROFILE}" -Ptest-patch It seems like we end up passing an extra empty string argument to the mvn call, and mvn interprets that as an extra goal to run, which of course doesn't exist. /opt/local/share/java/maven3/bin/mvn clean test -DskipTests -DhadoopPatchProcess -Ptest-patch > /tmp/hadoop-test-patch/2621/patchJavacWarnings.txt 2>&1
          Hide
          aw Allen Wittenauer added a comment -

          -00:

          • fix the native issue
          • add some detection for missing patchdirs
          • add a write_to_jira function to dedupe some code
          Show
          aw Allen Wittenauer added a comment - -00: fix the native issue add some detection for missing patchdirs add a write_to_jira function to dedupe some code
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6146//console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6146//console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
          -1 @author 0m 0s The patch appears to contain 1 @author tags which the Hadoop community has agreed to not allow in code contributions.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 release audit 0m 15s The applied patch does not increase the total number of release audit warnings.
          +1 shellcheck 0m 1s There were no new shellcheck (v) issues.
              0m 22s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12727176/HADOOP-11861-00.patch
          Optional Tests shellcheck
          git revision trunk / b08908a
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6146//console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. -1 @author 0m 0s The patch appears to contain 1 @author tags which the Hadoop community has agreed to not allow in code contributions. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 release audit 0m 15s The applied patch does not increase the total number of release audit warnings. +1 shellcheck 0m 1s There were no new shellcheck (v) issues.     0m 22s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727176/HADOOP-11861-00.patch Optional Tests shellcheck git revision trunk / b08908a Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6146//console This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          -01:

          • shellcheck binary detection was broken
          • shellcheck would sometimes report new errors that weren't because the lines changed. so now if the total count is the same we don't report problems.
          • @author checks for test-patch patches should get ignored since they should almost always have @authors in them for testing reasons.
          Show
          aw Allen Wittenauer added a comment - -01: shellcheck binary detection was broken shellcheck would sometimes report new errors that weren't because the lines changed. so now if the total count is the same we don't report problems. @author checks for test-patch patches should get ignored since they should almost always have @authors in them for testing reasons.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6148//console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6148//console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 release audit 0m 15s The applied patch does not increase the total number of release audit warnings.
          +1 shellcheck 0m 0s There were no new shellcheck (v) issues.
              0m 17s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12727241/HADOOP-11861-01.patch
          Optional Tests shellcheck
          git revision trunk / b08908a
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6148//console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 release audit 0m 15s The applied patch does not increase the total number of release audit warnings. +1 shellcheck 0m 0s There were no new shellcheck (v) issues.     0m 17s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727241/HADOOP-11861-01.patch Optional Tests shellcheck git revision trunk / b08908a Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6148//console This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          -02:

          • fix the console url
          • really, shellcheck should give +0 if the binary isn't available.
          Show
          aw Allen Wittenauer added a comment - -02: fix the console url really, shellcheck should give +0 if the binary isn't available.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6149//console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6149//console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 release audit 0m 14s The applied patch does not increase the total number of release audit warnings.
          0 shellcheck 0m 14s Shellcheck was not available.
              0m 17s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12727247/HADOOP-11861-02.patch
          Optional Tests shellcheck
          git revision trunk / b08908a
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6149/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 release audit 0m 14s The applied patch does not increase the total number of release audit warnings. 0 shellcheck 0m 14s Shellcheck was not available.     0m 17s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727247/HADOOP-11861-02.patch Optional Tests shellcheck git revision trunk / b08908a Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6149/console This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          -04: (vs -02):

          • better branch detection, esp for Jenkins and reexec mode
          • add --branch command line option
          • add better debugging for branch detection
          • force a pull after switching branches to make sure that branch is up-to-date
          Show
          aw Allen Wittenauer added a comment - -04: (vs -02): better branch detection, esp for Jenkins and reexec mode add --branch command line option add better debugging for branch detection force a pull after switching branches to make sure that branch is up-to-date
          Hide
          aw Allen Wittenauer added a comment -

          Chris Nauroth, please review and commit when you get a chance. It'd be better for this to go in sooner rather than later given the broken workspace bug that pops up on the Jenkins environment.

          Thanks.

          Show
          aw Allen Wittenauer added a comment - Chris Nauroth , please review and commit when you get a chance. It'd be better for this to go in sooner rather than later given the broken workspace bug that pops up on the Jenkins environment. Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6159//console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6159//console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 release audit 0m 16s The applied patch does not increase the total number of release audit warnings.
          0 shellcheck 0m 16s Shellcheck was not available.
              0m 19s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12727355/HADOOP-11861-04.patch
          Optional Tests shellcheck
          git revision trunk / 0ebe84d
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6159/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 release audit 0m 16s The applied patch does not increase the total number of release audit warnings. 0 shellcheck 0m 16s Shellcheck was not available.     0m 19s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727355/HADOOP-11861-04.patch Optional Tests shellcheck git revision trunk / 0ebe84d Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6159/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for patch v04. I did a test run with --build-native, and it worked fine. I committed this to trunk and branch-2.

          Anu, thank you for reporting the bug. Allen, thank you for providing the fixes.

          Show
          cnauroth Chris Nauroth added a comment - +1 for patch v04. I did a test run with --build-native , and it worked fine. I committed this to trunk and branch-2. Anu, thank you for reporting the bug. Allen, thank you for providing the fixes.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #7643 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7643/)
          HADOOP-11861. test-patch.sh rewrite addendum patch. Contributed by Allen Wittenauer. (cnauroth: rev 18eb5e79345295b2259b566c154375ad2a6216a1)

          • dev-support/test-patch.d/shellcheck.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • dev-support/test-patch.sh
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #7643 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7643/ ) HADOOP-11861 . test-patch.sh rewrite addendum patch. Contributed by Allen Wittenauer. (cnauroth: rev 18eb5e79345295b2259b566c154375ad2a6216a1) dev-support/test-patch.d/shellcheck.sh hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2104 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2104/)
          HADOOP-11861. test-patch.sh rewrite addendum patch. Contributed by Allen Wittenauer. (cnauroth: rev 18eb5e79345295b2259b566c154375ad2a6216a1)

          • dev-support/test-patch.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • dev-support/test-patch.d/shellcheck.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2104 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2104/ ) HADOOP-11861 . test-patch.sh rewrite addendum patch. Contributed by Allen Wittenauer. (cnauroth: rev 18eb5e79345295b2259b566c154375ad2a6216a1) dev-support/test-patch.sh hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.d/shellcheck.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #163 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/163/)
          HADOOP-11861. test-patch.sh rewrite addendum patch. Contributed by Allen Wittenauer. (cnauroth: rev 18eb5e79345295b2259b566c154375ad2a6216a1)

          • dev-support/test-patch.sh
          • dev-support/test-patch.d/shellcheck.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #163 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/163/ ) HADOOP-11861 . test-patch.sh rewrite addendum patch. Contributed by Allen Wittenauer. (cnauroth: rev 18eb5e79345295b2259b566c154375ad2a6216a1) dev-support/test-patch.sh dev-support/test-patch.d/shellcheck.sh hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #172 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/172/)
          HADOOP-11861. test-patch.sh rewrite addendum patch. Contributed by Allen Wittenauer. (cnauroth: rev 18eb5e79345295b2259b566c154375ad2a6216a1)

          • dev-support/test-patch.d/shellcheck.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • dev-support/test-patch.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #172 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/172/ ) HADOOP-11861 . test-patch.sh rewrite addendum patch. Contributed by Allen Wittenauer. (cnauroth: rev 18eb5e79345295b2259b566c154375ad2a6216a1) dev-support/test-patch.d/shellcheck.sh hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.sh
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #906 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/906/)
          HADOOP-11861. test-patch.sh rewrite addendum patch. Contributed by Allen Wittenauer. (cnauroth: rev 18eb5e79345295b2259b566c154375ad2a6216a1)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • dev-support/test-patch.sh
          • dev-support/test-patch.d/shellcheck.sh
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #906 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/906/ ) HADOOP-11861 . test-patch.sh rewrite addendum patch. Contributed by Allen Wittenauer. (cnauroth: rev 18eb5e79345295b2259b566c154375ad2a6216a1) hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.sh dev-support/test-patch.d/shellcheck.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #173 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/173/)
          HADOOP-11861. test-patch.sh rewrite addendum patch. Contributed by Allen Wittenauer. (cnauroth: rev 18eb5e79345295b2259b566c154375ad2a6216a1)

          • dev-support/test-patch.sh
          • dev-support/test-patch.d/shellcheck.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #173 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/173/ ) HADOOP-11861 . test-patch.sh rewrite addendum patch. Contributed by Allen Wittenauer. (cnauroth: rev 18eb5e79345295b2259b566c154375ad2a6216a1) dev-support/test-patch.sh dev-support/test-patch.d/shellcheck.sh hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2122 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2122/)
          HADOOP-11861. test-patch.sh rewrite addendum patch. Contributed by Allen Wittenauer. (cnauroth: rev 18eb5e79345295b2259b566c154375ad2a6216a1)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • dev-support/test-patch.sh
          • dev-support/test-patch.d/shellcheck.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2122 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2122/ ) HADOOP-11861 . test-patch.sh rewrite addendum patch. Contributed by Allen Wittenauer. (cnauroth: rev 18eb5e79345295b2259b566c154375ad2a6216a1) hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.sh dev-support/test-patch.d/shellcheck.sh

            People

            • Assignee:
              aw Allen Wittenauer
              Reporter:
              anu Anu Engineer
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development