Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12149

copy all of test-patch BINDIR prior to re-exec

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: HADOOP-12111, 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: yetus
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      During some tests (e.g., https://builds.apache.org/job/PreCommit-HADOOP-Build/7090 ), initial mvn install triggered a full test suite run when Jenkins switches from old test-patch to new test-patch. This is bad.

      1. HADOOP-12149.00.patch
        0.6 kB
        Allen Wittenauer

        Issue Links

          Activity

          Hide
          aw Allen Wittenauer added a comment - - edited

          Actually, it looks like the --project isn't getting picked up by Jenkins or making it past the reexec. So something else is wrong.

          Show
          aw Allen Wittenauer added a comment - - edited Actually, it looks like the --project isn't getting picked up by Jenkins or making it past the reexec. So something else is wrong.
          Hide
          aw Allen Wittenauer added a comment - - edited

          I sent a few tests through the system to the point that I'm pretty confident what is going on.

          It's a limitation of the old test-patch. It doesn't know about personalities so doesn't copy the personality directory during the re-exec. It only copies test-patch* and smart-apply-patch*.

          So there's not a bug here to fix, per se. We might default test-patch's personality setting to hadoop for the time being I guess?

          Show
          aw Allen Wittenauer added a comment - - edited I sent a few tests through the system to the point that I'm pretty confident what is going on. It's a limitation of the old test-patch. It doesn't know about personalities so doesn't copy the personality directory during the re-exec. It only copies test-patch* and smart-apply-patch*. So there's not a bug here to fix, per se. We might default test-patch's personality setting to hadoop for the time being I guess?
          Hide
          aw Allen Wittenauer added a comment - - edited

          Actually, that's not good enough because, again, there is no personality file to pick up. We need to hard-code hadoop's personality inside the built-in personality. Or modify trunk's test-patch to copy the entire directory.

          Show
          aw Allen Wittenauer added a comment - - edited Actually, that's not good enough because, again, there is no personality file to pick up. We need to hard-code hadoop's personality inside the built-in personality. Or modify trunk's test-patch to copy the entire directory.
          Hide
          busbey Sean Busbey added a comment -

          Would switching the Hadoop jenkins builds to use the yetus branch for test-patch instead of the one in the project under test solve this?

          At least while we're still working out of a branch in the repository and using the common-dev mail list, I think we are safely in the definition of Hadoop being "in the development community." (presuming the Hadoop PMC is fine with the switch, of course).

          Show
          busbey Sean Busbey added a comment - Would switching the Hadoop jenkins builds to use the yetus branch for test-patch instead of the one in the project under test solve this? At least while we're still working out of a branch in the repository and using the common-dev mail list, I think we are safely in the definition of Hadoop being "in the development community." (presuming the Hadoop PMC is fine with the switch, of course).
          Hide
          aw Allen Wittenauer added a comment -

          -00:

          • copy via tar the entire dev-support directory in the trunk version of test-patch
          Show
          aw Allen Wittenauer added a comment - -00: copy via tar the entire dev-support directory in the trunk version of test-patch
          Hide
          aw Allen Wittenauer added a comment -

          If we switch HADOOP over to use Yetus, we need to be all hands on deck to fix issues. And frankly, I'd rather not do that a few days before a holiday. That said, it would enable us to start using the docker mode and fix a ton of other bugs that people are reporting.

          On the flip side, modifying trunk's test-patch to copy over everything is a relatively tiny change. See -00.

          Show
          aw Allen Wittenauer added a comment - If we switch HADOOP over to use Yetus, we need to be all hands on deck to fix issues. And frankly, I'd rather not do that a few days before a holiday. That said, it would enable us to start using the docker mode and fix a ton of other bugs that people are reporting. On the flip side, modifying trunk's test-patch to copy over everything is a relatively tiny change. See -00.
          Hide
          busbey Sean Busbey added a comment -

          sounds reasonable. +1 (non-binding?) on the trunk change in 00.

          Show
          busbey Sean Busbey added a comment - sounds reasonable. +1 (non-binding?) on the trunk change in 00.
          Hide
          aw Allen Wittenauer added a comment -

          yeah, probably non-binding. Sorry.

          I'll activate the yetus signal to summon Chris Nauroth.

                       $Z$$$ZO
                     787$$ZZOO8O:    +
                    OZOOZ$+IZ$$OZD77I7?
                   $Z$$777$Z7$$ZZ$ZZ?77:
                  :7$7III$77$$I$7OZZ??77
                  O$7I7I?77II??7$OZO??77
                  O7III??ZI7II7$OOZ$+?IIO
                 8ZZZ$I$I+=I+$Z$OO$?++?IZ
                 O$7ZI,~II7IIZ7ZZ$?==++?7=
                 OZ77$,~:7$77+$Z$I====+?IZ
                 8$ZI??I7???I777?======?IO
                 ZZI7+??==+I77I====~===+IZ
                 ZZI?+~==+II7I=====~===+7$
                  I+==~?=~III+~===~~~=++7Z
                  $?==+~+III?=~===~~=+++$7
                   ?+I===I7I=~~~==~~=+??$
                   ??7?=III?~~~==~===+??$
                   7$O7=7I?=~~~~===++?+7
                    $O$=?I?~~~~===+????O
                    IZO=?I?~~~~=++????Z
                     IO+??=~===+???I?$O
                      8++?=++???IIII7O
                       7?? :+?II?II7O
                       8I?  ==???I$O
                        ?     =I7ZO
                                ?
          
          Show
          aw Allen Wittenauer added a comment - yeah, probably non-binding. Sorry. I'll activate the yetus signal to summon Chris Nauroth . $Z$$$ZO 787$$ZZOO8O: + OZOOZ$+IZ$$OZD77I7? $Z$$777$Z7$$ZZ$ZZ?77: :7$7III$77$$I$7OZZ??77 O$7I7I?77II??7$OZO??77 O7III??ZI7II7$OOZ$+?IIO 8ZZZ$I$I+=I+$Z$OO$?++?IZ O$7ZI,~II7IIZ7ZZ$?==++?7= OZ77$,~:7$77+$Z$I====+?IZ 8$ZI??I7???I777?======?IO ZZI7+??==+I77I====~===+IZ ZZI?+~==+II7I=====~===+7$ I+==~?=~III+~===~~~=++7Z $?==+~+III?=~===~~=+++$7 ?+I===I7I=~~~==~~=+??$ ??7?=III?~~~==~===+??$ 7$O7=7I?=~~~~===++?+7 $O$=?I?~~~~===+????O IZO=?I?~~~~=++????Z IO+??=~===+???I?$O 8++?=++???IIII7O 7?? :+?II?II7O 8I? ==???I$O ? =I7ZO ?
          Hide
          cnauroth Chris Nauroth added a comment -

          Is that what a Yetus looks like? Yikes!

          +1 for the patch.

          Show
          cnauroth Chris Nauroth added a comment - Is that what a Yetus looks like? Yikes! +1 for the patch.
          Hide
          aw Allen Wittenauer added a comment -

          lol

          thanks. committing to trunk.

          Show
          aw Allen Wittenauer added a comment - lol thanks. committing to trunk.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the files used for the QA process has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7113/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the files used for the QA process has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7113/console in case of problems.
          Hide
          aw Allen Wittenauer added a comment -
          • fired off a jenkins run, even though I've done it in my test jira just for completeness here.
          • updated summary to reflect the fix
          • moved to a full issue of Hadoop rather than a subtask for yetus
          Show
          aw Allen Wittenauer added a comment - fired off a jenkins run, even though I've done it in my test jira just for completeness here. updated summary to reflect the fix moved to a full issue of Hadoop rather than a subtask for yetus
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 release audit 0m 14s The applied patch does not increase the total number of release audit warnings.
          +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              0m 26s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12742876/HADOOP-12149.00.patch
          Optional Tests shellcheck
          git revision trunk / 076948d
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7113/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 release audit 0m 14s The applied patch does not increase the total number of release audit warnings. +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     0m 26s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742876/HADOOP-12149.00.patch Optional Tests shellcheck git revision trunk / 076948d Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7113/console This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          FYI, confirming that hadoop personality is now getting picked up. Hooray!

          Prior to this patch, the initial mvn install ran with:

          /home/jenkins/tools/maven/latest/bin/mvn --batch-mode -fae clean install -Dmaven.javadoc.skip=true -Ptest-patch > /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/branch-mvninstall-root.txt 2>&1
          

          post-this patch, it now runs with:

          /home/jenkins/tools/maven/latest/bin/mvn --batch-mode -fae clean install -Dmaven.javadoc.skip=true -DskipTests -Pnative -Drequire.snappy -Drequire.openssl -Drequire.fuse -Drequire.test.libhadoop -Ptest-patch > /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/branch-mvninstall-root.txt 2>&1
          

          Needless to say, skipping the unit tests is a huge speed up.

          Show
          aw Allen Wittenauer added a comment - FYI, confirming that hadoop personality is now getting picked up. Hooray! Prior to this patch, the initial mvn install ran with: /home/jenkins/tools/maven/latest/bin/mvn --batch-mode -fae clean install -Dmaven.javadoc.skip= true -Ptest-patch > /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/branch-mvninstall-root.txt 2>&1 post-this patch, it now runs with: /home/jenkins/tools/maven/latest/bin/mvn --batch-mode -fae clean install -Dmaven.javadoc.skip= true -DskipTests -Pnative -Drequire.snappy -Drequire.openssl -Drequire.fuse -Drequire.test.libhadoop -Ptest-patch > /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/branch-mvninstall-root.txt 2>&1 Needless to say, skipping the unit tests is a huge speed up.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8095 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8095/)
          HADOOP-12149. copy all of test-patch BINDIR prior to re-exec (aw) (aw: rev 147e020c7aef3ba42eddcef3be1b4ae7c7910371)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • dev-support/test-patch.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8095 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8095/ ) HADOOP-12149 . copy all of test-patch BINDIR prior to re-exec (aw) (aw: rev 147e020c7aef3ba42eddcef3be1b4ae7c7910371) hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #975 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/975/)
          HADOOP-12149. copy all of test-patch BINDIR prior to re-exec (aw) (aw: rev 147e020c7aef3ba42eddcef3be1b4ae7c7910371)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • dev-support/test-patch.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #975 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/975/ ) HADOOP-12149 . copy all of test-patch BINDIR prior to re-exec (aw) (aw: rev 147e020c7aef3ba42eddcef3be1b4ae7c7910371) hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.sh
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #245 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/245/)
          HADOOP-12149. copy all of test-patch BINDIR prior to re-exec (aw) (aw: rev 147e020c7aef3ba42eddcef3be1b4ae7c7910371)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • dev-support/test-patch.sh
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #245 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/245/ ) HADOOP-12149 . copy all of test-patch BINDIR prior to re-exec (aw) (aw: rev 147e020c7aef3ba42eddcef3be1b4ae7c7910371) hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2172 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2172/)
          HADOOP-12149. copy all of test-patch BINDIR prior to re-exec (aw) (aw: rev 147e020c7aef3ba42eddcef3be1b4ae7c7910371)

          • dev-support/test-patch.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2172 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2172/ ) HADOOP-12149 . copy all of test-patch BINDIR prior to re-exec (aw) (aw: rev 147e020c7aef3ba42eddcef3be1b4ae7c7910371) dev-support/test-patch.sh hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #233 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/233/)
          HADOOP-12149. copy all of test-patch BINDIR prior to re-exec (aw) (aw: rev 147e020c7aef3ba42eddcef3be1b4ae7c7910371)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • dev-support/test-patch.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #233 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/233/ ) HADOOP-12149 . copy all of test-patch BINDIR prior to re-exec (aw) (aw: rev 147e020c7aef3ba42eddcef3be1b4ae7c7910371) hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2191 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2191/)
          HADOOP-12149. copy all of test-patch BINDIR prior to re-exec (aw) (aw: rev 147e020c7aef3ba42eddcef3be1b4ae7c7910371)

          • dev-support/test-patch.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2191 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2191/ ) HADOOP-12149 . copy all of test-patch BINDIR prior to re-exec (aw) (aw: rev 147e020c7aef3ba42eddcef3be1b4ae7c7910371) dev-support/test-patch.sh hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #243 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/243/)
          HADOOP-12149. copy all of test-patch BINDIR prior to re-exec (aw) (aw: rev 147e020c7aef3ba42eddcef3be1b4ae7c7910371)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • dev-support/test-patch.sh
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #243 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/243/ ) HADOOP-12149 . copy all of test-patch BINDIR prior to re-exec (aw) (aw: rev 147e020c7aef3ba42eddcef3be1b4ae7c7910371) hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.sh

            People

            • Assignee:
              aw Allen Wittenauer
              Reporter:
              aw Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development