Hadoop Common
  1. Hadoop Common
  2. HADOOP-7167

Allow using a file to exclude certain tests from build

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

      1. hadoop-7167-cygwin.txt
        0.9 kB
        Todd Lipcon
      2. hadoop-7167.txt
        1 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment -

          +1 patch looks good. With only exception that we shouldn't overuse this 'facility'

          Show
          Konstantin Boudnik added a comment - +1 patch looks good. With only exception that we shouldn't overuse this 'facility'
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #624 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/624/)
          HADOOP-7167. Allow using a file to exclude certain tests from build. Contributed by Todd Lipcon.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #624 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/624/ ) HADOOP-7167 . Allow using a file to exclude certain tests from build. Contributed by Todd Lipcon.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Todd, the exclude file does not work on Cygwin. Could you take a look?

          ...
          BUILD FAILED
          z:\_tsz\hadoop\hdfs\h1\build.xml:745: The following error occurred while executing this line:
          z:\_tsz\hadoop\hdfs\h1\build.xml:677: The following error occurred while executing this line:
          z:\_tsz\hadoop\hdfs\h1\build.xml:613: Excludesfile z:\dev\null not found.
          
          Total time: 16 seconds
          
          Show
          Tsz Wo Nicholas Sze added a comment - Hi Todd, the exclude file does not work on Cygwin. Could you take a look? ... BUILD FAILED z:\_tsz\hadoop\hdfs\h1\build.xml:745: The following error occurred while executing this line: z:\_tsz\hadoop\hdfs\h1\build.xml:677: The following error occurred while executing this line: z:\_tsz\hadoop\hdfs\h1\build.xml:613: Excludesfile z:\dev\null not found. Total time: 16 seconds
          Hide
          Todd Lipcon added a comment -

          Hey Nicholas. Really sorry about that! Do you know if there is some equivalent to /dev/null in cygwin? ie a file which is always guaranteed to exist and be empty? I wonder what happens if we just pass an empty parameter to the ant excludeFile tag...

          If this is getting in your way, feel free to revert and we can commit a new version that works on both platforms.

          Show
          Todd Lipcon added a comment - Hey Nicholas. Really sorry about that! Do you know if there is some equivalent to /dev/null in cygwin? ie a file which is always guaranteed to exist and be empty? I wonder what happens if we just pass an empty parameter to the ant excludeFile tag... If this is getting in your way, feel free to revert and we can commit a new version that works on both platforms.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Cygwin actually has /dev/null. The following works.

          $ ls > /dev/null
          

          Interestingly, ls shows it only if the full path is given.

          $ ls -l /dev/null
          crw-rw-rw- 1 tsz Users 1, 3 Nov 30  2006 /dev/null
          
          $ ls -l /dev
          total 4
          lrwxrwxrwx 1 tsz Users 13 Oct 23  2009 fd -> /proc/self/fd
          lrwxrwxrwx 1 tsz Users 15 Oct 23  2009 stderr -> /proc/self/fd/2
          lrwxrwxrwx 1 tsz Users 15 Oct 23  2009 stdin -> /proc/self/fd/0
          lrwxrwxrwx 1 tsz Users 15 Oct 23  2009 stdout -> /proc/self/fd/1
          
          Show
          Tsz Wo Nicholas Sze added a comment - Cygwin actually has /dev/null . The following works. $ ls > /dev/null Interestingly, ls shows it only if the full path is given. $ ls -l /dev/null crw-rw-rw- 1 tsz Users 1, 3 Nov 30 2006 /dev/null $ ls -l /dev total 4 lrwxrwxrwx 1 tsz Users 13 Oct 23 2009 fd -> /proc/self/fd lrwxrwxrwx 1 tsz Users 15 Oct 23 2009 stderr -> /proc/self/fd/2 lrwxrwxrwx 1 tsz Users 15 Oct 23 2009 stdin -> /proc/self/fd/0 lrwxrwxrwx 1 tsz Users 15 Oct 23 2009 stdout -> /proc/self/fd/1
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/523/)

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/523/ )
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Todd, have you got a chance to take a look? In case you don't have time to work on this, could you revert this, HDFS-1731 and MAPREDUCE-2367? Many thanks.

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Todd, have you got a chance to take a look? In case you don't have time to work on this, could you revert this, HDFS-1731 and MAPREDUCE-2367 ? Many thanks.
          Hide
          Todd Lipcon added a comment -

          Yes, sorry - I am planning to add an empty file in "src/test/empty-file" and use that as the default instead of /dev/null. Sound good?

          Show
          Todd Lipcon added a comment - Yes, sorry - I am planning to add an empty file in "src/test/empty-file" and use that as the default instead of /dev/null. Sound good?
          Hide
          Todd Lipcon added a comment -

          How does this patch look?

          Show
          Todd Lipcon added a comment - How does this patch look?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The patch works if I create src/test/empty-file manually.

          BUILD FAILED
          z:\_tsz\hadoop\common\c2\build.xml:827: The following error occurred while executing this line:
          z:\_tsz\hadoop\common\c2\build.xml:807: The following error occurred while executing this line:
          z:\_tsz\hadoop\common\c2\build.xml:736: Excludesfile z:\_tsz\hadoop\common\c2\src\test\empty-file not found.
          
          Total time: 22 seconds
          
          $ touch src/test/empty-file
          
          ...
          BUILD SUCCESSFUL
          Total time: 1 minute 4 seconds
          
          Show
          Tsz Wo Nicholas Sze added a comment - The patch works if I create src/test/empty-file manually. BUILD FAILED z:\_tsz\hadoop\common\c2\build.xml:827: The following error occurred while executing this line: z:\_tsz\hadoop\common\c2\build.xml:807: The following error occurred while executing this line: z:\_tsz\hadoop\common\c2\build.xml:736: Excludesfile z:\_tsz\hadoop\common\c2\src\test\empty-file not found. Total time: 22 seconds $ touch src/test/empty-file ... BUILD SUCCESSFUL Total time: 1 minute 4 seconds
          Hide
          Todd Lipcon added a comment -

          cool, +1 for commit then? (including the new empty-file)

          Show
          Todd Lipcon added a comment - cool, +1 for commit then? (including the new empty-file)
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good, thanks!

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good, thanks!
          Hide
          Todd Lipcon added a comment -

          Commited fix to trunk. Will go and add patches to the hdfs and mr JIRAs in just a minute.

          Show
          Todd Lipcon added a comment - Commited fix to trunk. Will go and add patches to the hdfs and mr JIRAs in just a minute.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #527 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/527/)
          HADOOP-7167. Amend previous commit under this JIRA to fix issue on cygwin. Contributed by Todd Lipcon

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #527 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/527/ ) HADOOP-7167 . Amend previous commit under this JIRA to fix issue on cygwin. Contributed by Todd Lipcon
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #628 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/628/)
          HADOOP-7167. Amend previous commit under this JIRA to fix issue on cygwin. Contributed by Todd Lipcon

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #628 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/628/ ) HADOOP-7167 . Amend previous commit under this JIRA to fix issue on cygwin. Contributed by Todd Lipcon
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #539 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/539/)
          Remove redundant entry for HADOOP-7167 in CHANGES.txt

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #539 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/539/ ) Remove redundant entry for HADOOP-7167 in CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #647 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/647/)
          Remove redundant entry for HADOOP-7167 in CHANGES.txt

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #647 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/647/ ) Remove redundant entry for HADOOP-7167 in CHANGES.txt

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development