Hadoop Common
  1. Hadoop Common
  2. HADOOP-7291

Update Hudson job not to run test-contrib

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.

      1. HADOOP-7291-3.patch
        0.5 kB
        Eli Collins
      2. HADOOP-7291-2.patch
        2 kB
        Eli Collins
      3. HADOOP-7291.patch
        3 kB
        Nigel Daley

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-22-branch #49 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/49/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #49 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/49/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #682 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/682/)
          HADOOP-7291. Remove spurious call to runTestContrib. Contributed by Eli Collins
          MAPREDUCE-2499. MR part of HADOOP-7291. Contributed by Eli Collins
          HADOOP-7291. Update Hudson job not to run test-contrib. Contributed by Nigel Daley
          Reverting the change r1102914 for HADOOP-7291 to fix build issues.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #682 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/682/ ) HADOOP-7291 . Remove spurious call to runTestContrib. Contributed by Eli Collins MAPREDUCE-2499 . MR part of HADOOP-7291 . Contributed by Eli Collins HADOOP-7291 . Update Hudson job not to run test-contrib. Contributed by Nigel Daley Reverting the change r1102914 for HADOOP-7291 to fix build issues.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-22-branch #52 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/52/)
          HADOOP-7291. svn merge -c 1103931 from trunk

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #52 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/52/ ) HADOOP-7291 . svn merge -c 1103931 from trunk
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #691 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/691/)
          HADOOP-7291. Remove spurious call to runTestContrib. Contributed by Eli Collins
          HADOOP-7291. Update Hudson job not to run test-contrib. Contributed by Nigel Daley
          Reverting the change r1102914 for HADOOP-7291 to fix build issues.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #691 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/691/ ) HADOOP-7291 . Remove spurious call to runTestContrib. Contributed by Eli Collins HADOOP-7291 . Update Hudson job not to run test-contrib. Contributed by Nigel Daley Reverting the change r1102914 for HADOOP-7291 to fix build issues.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-22-branch #49 (See https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/49/)
          HADOOP-7291. svn merge -c 1103931 from trunk

          Show
          Hudson added a comment - Integrated in Hadoop-Common-22-branch #49 (See https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/49/ ) HADOOP-7291 . svn merge -c 1103931 from trunk
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #658 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/658/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #658 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/658/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #677 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/677/)
          HADOOP-7291. Remove spurious call to runTestContrib. Contributed by Eli Collins

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #677 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/677/ ) HADOOP-7291 . Remove spurious call to runTestContrib. Contributed by Eli Collins
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #602 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/602/)
          HADOOP-7291. Remove spurious call to runTestContrib. Contributed by Eli Collins

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #602 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/602/ ) HADOOP-7291 . Remove spurious call to runTestContrib. Contributed by Eli Collins
          Hide
          Eli Collins added a comment -

          Thanks Todd. I've committed this to trunk and 22. Verified that the precommit Hudson job runs now.

          Show
          Eli Collins added a comment - Thanks Todd. I've committed this to trunk and 22. Verified that the precommit Hudson job runs now.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479396/HADOOP-7291-3.patch
          against trunk revision 1103931.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/458//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479396/HADOOP-7291-3.patch against trunk revision 1103931. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/458//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          +1 for HADOOP-7291-3.patch

          Show
          Todd Lipcon added a comment - +1 for HADOOP-7291 -3.patch
          Hide
          Eli Collins added a comment -

          After HDFS-1946 and MR-2499 the common, HDFS and MR precommit jobs run though looks like $GREP is not defined. This is because a new call to runContribTests was introduced in HADOOP-7291.patch which runs before parseArgs. We don't need this call. The attached patch removes it.

          Show
          Eli Collins added a comment - After HDFS-1946 and MR-2499 the common, HDFS and MR precommit jobs run though looks like $GREP is not defined. This is because a new call to runContribTests was introduced in HADOOP-7291 .patch which runs before parseArgs. We don't need this call. The attached patch removes it.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #676 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/676/)
          MAPREDUCE-2499. MR part of HADOOP-7291. Contributed by Eli Collins

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #676 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/676/ ) MAPREDUCE-2499 . MR part of HADOOP-7291 . Contributed by Eli Collins
          Hide
          Eli Collins added a comment -

          The HDFS and MR build.xml files need to be updated to not pass python.home.

          Show
          Eli Collins added a comment - The HDFS and MR build.xml files need to be updated to not pass python.home.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #675 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/675/)
          HADOOP-7291. Update Hudson job not to run test-contrib. Contributed by Nigel Daley

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #675 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/675/ ) HADOOP-7291 . Update Hudson job not to run test-contrib. Contributed by Nigel Daley
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #601 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/601/)
          HADOOP-7291. Update Hudson job not to run test-contrib. Contributed by Nigel Daley

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #601 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/601/ ) HADOOP-7291 . Update Hudson job not to run test-contrib. Contributed by Nigel Daley
          Hide
          Eli Collins added a comment -

          I committed the patch and Nigel updated the Hudson job.

          I filed HADOOP-7295 so this could be done in a single commit w/o breaking Hudson.

          Show
          Eli Collins added a comment - I committed the patch and Nigel updated the Hudson job. I filed HADOOP-7295 so this could be done in a single commit w/o breaking Hudson.
          Hide
          Eli Collins added a comment -

          Nigel and I discussed. We need to:

          1. Apply the HADOOP-7291.patch again
          2. Update the Hudson job to not pass the python argument

          I'm doing #1, Nigel is doing #2.

          Apologies for the disruption guys!

          Show
          Eli Collins added a comment - Nigel and I discussed. We need to: Apply the HADOOP-7291 .patch again Update the Hudson job to not pass the python argument I'm doing #1, Nigel is doing #2. Apologies for the disruption guys!
          Hide
          Suresh Srinivas added a comment -

          I have reverted the patch for now.

          Show
          Suresh Srinivas added a comment - I have reverted the patch for now.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > ... This is urgent, since pre-commit builds cannot be run.

          How about we temporarily revert the committed patch?

          Show
          Tsz Wo Nicholas Sze added a comment - > ... This is urgent, since pre-commit builds cannot be run. How about we temporarily revert the committed patch?
          Hide
          Giridharan Kesavan added a comment -

          Right, looking at the log more closely ("-c: command not found") it looks like GREP is not defined. How is test-patch being called?

          test-patch is called through ant build.xml hudson-test-patch target.
          $

          {ANT_HOME}

          /bin/ant \
          -Dpatch.file=foobar \
          -Dscratch.dir=$

          {WORKSPACE}

          /patchprocess \
          -Dsupport.dir=/homes/hudson/buildSupport \
          -Dps.cmd=/bin/ps \
          -Dwget.cmd=/usr/bin/wget \
          -Djiracli.cmd=/homes/hudson/tools/jiracli-1.5/jira \
          -Dsvn.cmd=/usr/bin/svn \
          -Dgrep.cmd=/bin/grep \
          -Dpatch.cmd=/usr/bin/patch \
          -Dfindbugs.home=/homes/hudson/tools/findbugs/latest \
          -Dforrest.home=/homes/hudson/tools/forrest/latest \
          -Declipse.home=/homes/hudson/tools/eclipse/latest \
          -Dpython.home=/homes/hudson/tools/python/latest \
          -Djira.passwd=<> \
          -Dcurl.cmd=/usr/bin/curl \
          Ddefect=HDFS$

          {ISSUE_NUM}

          \
          hudson-test-patch

          Show
          Giridharan Kesavan added a comment - Right, looking at the log more closely ("-c: command not found") it looks like GREP is not defined. How is test-patch being called? test-patch is called through ant build.xml hudson-test-patch target. $ {ANT_HOME} /bin/ant \ -Dpatch.file=foobar \ -Dscratch.dir=$ {WORKSPACE} /patchprocess \ -Dsupport.dir=/homes/hudson/buildSupport \ -Dps.cmd=/bin/ps \ -Dwget.cmd=/usr/bin/wget \ -Djiracli.cmd=/homes/hudson/tools/jiracli-1.5/jira \ -Dsvn.cmd=/usr/bin/svn \ -Dgrep.cmd=/bin/grep \ -Dpatch.cmd=/usr/bin/patch \ -Dfindbugs.home=/homes/hudson/tools/findbugs/latest \ -Dforrest.home=/homes/hudson/tools/forrest/latest \ -Declipse.home=/homes/hudson/tools/eclipse/latest \ -Dpython.home=/homes/hudson/tools/python/latest \ -Djira.passwd=<> \ -Dcurl.cmd=/usr/bin/curl \ Ddefect=HDFS $ {ISSUE_NUM} \ hudson-test-patch
          Hide
          Suresh Srinivas added a comment -

          I see hudson pre-commit builds failing with this:
          [exec] Total time: 4 seconds
          [exec] ERROR: usage /grid/0/hudson/hudson-slave/workspace/PreCommit-HDFS-Build/trunk/src/test/bin/test-patch.sh HUDSON <PATCH_DIR> <SUPPORT_DIR> <PS_CMD> <WGET_CMD> <JIRACLI> <SVN_CMD> <GREP_CMD> <PATCH_CMD> <FINDBUGS_HOME> <FORREST_HOME> <ECLIPSE_HOME> <WORKSPACE_BASEDIR> <JIRA_PASSWD> <CURL_CMD> <DEFECT>

          Eli or Nigel, could you please take a look at this? This is urgent, since pre-commit builds cannot be run.

          Show
          Suresh Srinivas added a comment - I see hudson pre-commit builds failing with this: [exec] Total time: 4 seconds [exec] ERROR: usage /grid/0/hudson/hudson-slave/workspace/PreCommit-HDFS-Build/trunk/src/test/bin/test-patch.sh HUDSON <PATCH_DIR> <SUPPORT_DIR> <PS_CMD> <WGET_CMD> <JIRACLI> <SVN_CMD> <GREP_CMD> <PATCH_CMD> <FINDBUGS_HOME> <FORREST_HOME> <ECLIPSE_HOME> <WORKSPACE_BASEDIR> <JIRA_PASSWD> <CURL_CMD> <DEFECT> Eli or Nigel, could you please take a look at this? This is urgent, since pre-commit builds cannot be run.
          Hide
          Giridharan Kesavan added a comment -

          https://builds.apache.org/hudson/job/PreCommit-Hdfs-Build/524/console

          This is breaking all the test-patch build of hdfs

          Show
          Giridharan Kesavan added a comment - https://builds.apache.org/hudson/job/PreCommit-Hdfs-Build/524/console This is breaking all the test-patch build of hdfs
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #679 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/679/)
          HADOOP-7291. Update Hudson job not to run test-contrib. Contributed by Nigel Daley

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #679 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/679/ ) HADOOP-7291 . Update Hudson job not to run test-contrib. Contributed by Nigel Daley
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #688 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/688/)
          HADOOP-7291. Update Hudson job not to run test-contrib. Contributed by Nigel Daley

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #688 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/688/ ) HADOOP-7291 . Update Hudson job not to run test-contrib. Contributed by Nigel Daley
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-22-branch #51 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/51/)
          HADOOP-7291. svn merge -c 1102914 from trunk

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #51 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/51/ ) HADOOP-7291 . svn merge -c 1102914 from trunk
          Hide
          Eli Collins added a comment -

          Right, looking at the log more closely ("-c: command not found") it looks like GREP is not defined. How is test-patch being called?

          Show
          Eli Collins added a comment - Right, looking at the log more closely ("-c: command not found") it looks like GREP is not defined. How is test-patch being called?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Eli, test-patch is also used in HDFS and MapReduce. So, we cannot remove test-contrib.

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Eli, test-patch is also used in HDFS and MapReduce. So, we cannot remove test-contrib.
          Hide
          Eli Collins added a comment -

          Patch attached. Since there's no test-contrib target we can just remove it entirely.

          Show
          Eli Collins added a comment - Patch attached. Since there's no test-contrib target we can just remove it entirely.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          In build #454, there is a build failed.

               [exec] ======================================================================
               [exec] /grid/0/hudson/hudson-slave/workspace/PreCommit-HADOOP-Build/trunk/src/test/bin/test-patch.sh: line 527: -c: command not found
               [exec] ======================================================================
               [exec]     Running contrib tests.
               [exec] ======================================================================
               [exec] ======================================================================
               [exec] 
               [exec] 
               [exec] /grid/0/hudson/hudson-slave/workspace/PreCommit-HADOOP-Build/trunk/src/test/bin/test-patch.sh: line 533: auxwww: command not found
               [exec] /grid/0/hudson/hudson-slave/workspace/PreCommit-HADOOP-Build/trunk/src/test/bin/test-patch.sh: line 533: HadoopPatchProcess: command not found
               [exec] /homes/hudson/tools/ant/latest/bin/ant -Dversion=  -DHadoopPatchProcess= -Dtest.junit.output.format=xml -Dtest.output=no test-contrib
               [exec] Buildfile: build.xml
               [exec] 
               [exec] BUILD FAILED
               [exec] Target "test-contrib" does not exist in the project "Hadoop-Common". 
               [exec] 
               [exec] Total time: 0 seconds
          
          Show
          Tsz Wo Nicholas Sze added a comment - In build #454 , there is a build failed. [exec] ====================================================================== [exec] /grid/0/hudson/hudson-slave/workspace/PreCommit-HADOOP-Build/trunk/src/test/bin/test-patch.sh: line 527: -c: command not found [exec] ====================================================================== [exec] Running contrib tests. [exec] ====================================================================== [exec] ====================================================================== [exec] [exec] [exec] /grid/0/hudson/hudson-slave/workspace/PreCommit-HADOOP-Build/trunk/src/test/bin/test-patch.sh: line 533: auxwww: command not found [exec] /grid/0/hudson/hudson-slave/workspace/PreCommit-HADOOP-Build/trunk/src/test/bin/test-patch.sh: line 533: HadoopPatchProcess: command not found [exec] /homes/hudson/tools/ant/latest/bin/ant -Dversion= -DHadoopPatchProcess= -Dtest.junit.output.format=xml -Dtest.output=no test-contrib [exec] Buildfile: build.xml [exec] [exec] BUILD FAILED [exec] Target "test-contrib" does not exist in the project "Hadoop-Common". [exec] [exec] Total time: 0 seconds
          Hide
          Eli Collins added a comment -

          Forgot to mention, please enable precommit builds when you get a chance.

          Show
          Eli Collins added a comment - Forgot to mention, please enable precommit builds when you get a chance.
          Hide
          Eli Collins added a comment -

          I've committed this to trunk and branch 22. Thanks Nigel!

          Show
          Eli Collins added a comment - I've committed this to trunk and branch 22. Thanks Nigel!
          Hide
          Eli Collins added a comment -

          Ah. +1 patch lgtm

          Show
          Eli Collins added a comment - Ah. +1 patch lgtm
          Hide
          Nigel Daley added a comment -

          This patch includes removing Python dependency since HOD is gone. It also conditionally run test-contrib if the project has a test-contrib target in build.xml

          Remember that the test-patch.sh script is pulled from common into hdfs and mapreduce using svn:externals

          Show
          Nigel Daley added a comment - This patch includes removing Python dependency since HOD is gone. It also conditionally run test-contrib if the project has a test-contrib target in build.xml Remember that the test-patch.sh script is pulled from common into hdfs and mapreduce using svn:externals
          Hide
          Nigel Daley added a comment -

          No, test-contrib is called directly in src/test/bin/test-patch.sh

          I'll upload an untested patch in a sec.

          Show
          Nigel Daley added a comment - No, test-contrib is called directly in src/test/bin/test-patch.sh I'll upload an untested patch in a sec.
          Hide
          Eli Collins added a comment -

          Btw Nigel, if you can point me to the Hudson login I'm happy to update the job command.

          Show
          Eli Collins added a comment - Btw Nigel, if you can point me to the Hudson login I'm happy to update the job command.
          Hide
          Eli Collins added a comment -

          Apologies for missing this, I assumed test-contrib was being called by Hudson only via the dependency on the test target.

          Show
          Eli Collins added a comment - Apologies for missing this, I assumed test-contrib was being called by Hudson only via the dependency on the test target.
          Hide
          Nigel Daley added a comment -

          I've disabled further precommit builds until we fix this. I had partial change in my workspace for this very issue which I why I wasn't yet committing 7137. Sorry, should have noted that.

          Show
          Nigel Daley added a comment - I've disabled further precommit builds until we fix this. I had partial change in my workspace for this very issue which I why I wasn't yet committing 7137. Sorry, should have noted that.

            People

            • Assignee:
              Eli Collins
              Reporter:
              Eli Collins
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development