Lucene - Core
  1. Lucene - Core
  2. LUCENE-4406

Print out where tests failed at the end of running the Test Suite

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0, 6.0
    • Component/s: general/test
    • Labels:
      None

      Description

      It would be nice if, at the end of running ant test, it spit out the names of which tests failed so that one doesn't have to go scrolling up through the output or go run grep on the test-reports as a separate step.

      For another project, I use:

      <target name="test-summary">
          <echo>Looking for summaries in: ${build.dir}/test-reports with basedir: ${basedir}</echo>
          <echo>Errors:</echo>
          <exec executable="grep">
                  <arg value="-r"/>
                  <arg value="-rl"/>
                  <arg value="errors=\&quot;[1-9]\&quot;"/>
                  <arg value="${build.dir}/test-reports"/>
          </exec>
          <echo>Failures:</echo>
          <exec executable="grep">
                  <arg value="-r"/>
                  <arg value="-rl"/>
                  <arg value="failures=\&quot;[1-9]\&quot;"/>
                  <arg value="${build.dir}/test-reports"/>
          </exec>
        </target>
      

      which can likely be modified for Lucene. I can do it, but wanted to see if others had an opinion.

        Activity

        Hide
        Steve Rowe added a comment -

        +1

        Show
        Steve Rowe added a comment - +1
        Hide
        Dawid Weiss added a comment -

        There are a few different ways to do it. I can also modify the default text report to include such an information, or we could extract it from ANT test XMLs generated during the run (this can be misleading if there are existing XMLs in the output directory and one runs just a subset of tests). I wouldn't use "exec grep" because this isn't always available (windows machines). There is a task for Ant that does it I'm sure (or it can be built).

        I think the easiest way would be to include failed tests (or suites?) at the end via an attribute on the text report in randomized testing. Let me know if you'd want it done this way.

        Show
        Dawid Weiss added a comment - There are a few different ways to do it. I can also modify the default text report to include such an information, or we could extract it from ANT test XMLs generated during the run (this can be misleading if there are existing XMLs in the output directory and one runs just a subset of tests). I wouldn't use "exec grep" because this isn't always available (windows machines). There is a task for Ant that does it I'm sure (or it can be built). I think the easiest way would be to include failed tests (or suites?) at the end via an attribute on the text report in randomized testing. Let me know if you'd want it done this way.
        Hide
        Dawid Weiss added a comment -

        I've worked on the release today anyway to fix the problematic non-allowed XML characters in reports so I added this as well. It looks pretty much like this (the number of failures to show is configurable):

        [junit4:junit4]
        [junit4:junit4] Tests with failures (first 2 out of 6):
        [junit4:junit4]   - com.carrotsearch.ant.tasks.junit4.tests.TestStatuses.failure
        [junit4:junit4]   - com.carrotsearch.ant.tasks.junit4.tests.TestStatuses.error
        

        I didn't include the throwable's message like Maven does because it really doesn't make sense here – if you care about a failure you'll need the stack (and possibly sysouts) too so you'll have to locate it anyway. Btw. remember that all failures are also marked with a "<<<" so grepping for this in context may also provide a nice shortcut to locate the full failure string.

        Show
        Dawid Weiss added a comment - I've worked on the release today anyway to fix the problematic non-allowed XML characters in reports so I added this as well. It looks pretty much like this (the number of failures to show is configurable): [junit4:junit4] [junit4:junit4] Tests with failures (first 2 out of 6): [junit4:junit4] - com.carrotsearch.ant.tasks.junit4.tests.TestStatuses.failure [junit4:junit4] - com.carrotsearch.ant.tasks.junit4.tests.TestStatuses.error I didn't include the throwable's message like Maven does because it really doesn't make sense here – if you care about a failure you'll need the stack (and possibly sysouts) too so you'll have to locate it anyway. Btw. remember that all failures are also marked with a "<<<" so grepping for this in context may also provide a nice shortcut to locate the full failure string.
        Hide
        Steve Rowe added a comment -

        FYI, Dawid, Maven was able to download the new randomizedtesting-runner:2.0.1 dependency from the Sonatype OSS repo right after you committed this:

        [INFO] ------------------------------------------------------------------------
        [INFO] Building Lucene Test Framework
        [INFO]    task-segment: [install]
        [INFO] ------------------------------------------------------------------------
        Downloading: http://oss.sonatype.org/content/repositories/releases/com/carrotsearch/randomizedtesting/randomizedtesting-runner/2.0.1/randomizedtesting-runner-2.0.1.pom
              
        Downloading: http://oss.sonatype.org/content/repositories/releases/com/carrotsearch/randomizedtesting/randomizedtesting-parent/2.0.1/randomizedtesting-parent-2.0.1.pom
        
        Downloading: http://oss.sonatype.org/content/repositories/releases/com/carrotsearch/randomizedtesting/randomizedtesting-runner/2.0.1/randomizedtesting-runner-2.0.1.jar
        190K downloaded  (randomizedtesting-runner-2.0.1.jar)
        
        Show
        Steve Rowe added a comment - FYI, Dawid, Maven was able to download the new randomizedtesting-runner:2.0.1 dependency from the Sonatype OSS repo right after you committed this: [INFO] ------------------------------------------------------------------------ [INFO] Building Lucene Test Framework [INFO] task-segment: [install] [INFO] ------------------------------------------------------------------------ Downloading: http://oss.sonatype.org/content/repositories/releases/com/carrotsearch/randomizedtesting/randomizedtesting-runner/2.0.1/randomizedtesting-runner-2.0.1.pom Downloading: http://oss.sonatype.org/content/repositories/releases/com/carrotsearch/randomizedtesting/randomizedtesting-parent/2.0.1/randomizedtesting-parent-2.0.1.pom Downloading: http://oss.sonatype.org/content/repositories/releases/com/carrotsearch/randomizedtesting/randomizedtesting-runner/2.0.1/randomizedtesting-runner-2.0.1.jar 190K downloaded (randomizedtesting-runner-2.0.1.jar)
        Hide
        Uwe Schindler added a comment -

        But SHA1 che chksums are missing:

        check-licenses:
             [echo] License check under: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene
         [licenses] MISSING sha1 checksum file for: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/test-framework/lib/junit4-ant-2.0.1.jar
         [licenses] MISSING sha1 checksum file for: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/test-framework/lib/randomizedtesting-runner-2.0.1.jar
         [licenses] Scanned 19 JAR file(s) for licenses (in 0.26s.), 2 error(s).
        
        Show
        Uwe Schindler added a comment - But SHA1 che chksums are missing: check-licenses: [echo] License check under: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene [licenses] MISSING sha1 checksum file for: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/test-framework/lib/junit4-ant-2.0.1.jar [licenses] MISSING sha1 checksum file for: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/test-framework/lib/randomizedtesting-runner-2.0.1.jar [licenses] Scanned 19 JAR file(s) for licenses (in 0.26s.), 2 error(s).
        Hide
        Robert Muir added a comment -

        I'll run ant jar-checksums and fix this.

        Show
        Robert Muir added a comment - I'll run ant jar-checksums and fix this.
        Hide
        Dawid Weiss added a comment -

        Thanks guys, my bad – I actually ran ant precommit but I work on a git checkout and it failed on me (svn required). Last minute commit before leaving - red card.

        Show
        Dawid Weiss added a comment - Thanks guys, my bad – I actually ran ant precommit but I work on a git checkout and it failed on me (svn required). Last minute commit before leaving - red card.
        Hide
        Dawid Weiss added a comment -

        As for maven and sonatype - it shouldn't be a problem, I know, but this time I waited. It took 5 hours to sync between sonatype and maven central. I admit it is a pain to wait so long before you can make an update to downstream projects.

        Show
        Dawid Weiss added a comment - As for maven and sonatype - it shouldn't be a problem, I know, but this time I waited. It took 5 hours to sync between sonatype and maven central. I admit it is a pain to wait so long before you can make an update to downstream projects.
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Robert Muir
        http://svn.apache.org/viewvc?view=revision&revision=1387609

        LUCENE-4406: regenerate checksums (ant jar-checksums) for new randomizedtesting jars

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Robert Muir http://svn.apache.org/viewvc?view=revision&revision=1387609 LUCENE-4406 : regenerate checksums (ant jar-checksums) for new randomizedtesting jars
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Dawid Weiss
        http://svn.apache.org/viewvc?view=revision&revision=1387589

        LUCENE-4406, LUCENE-4407: Upgrade to randomizedtesting 2.0.1.
        Workaround for broken test output XMLs due to non-XML text unicode
        chars in strings. Added printing of failed tests at the end of a
        test run (Dawid Weiss)

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Dawid Weiss http://svn.apache.org/viewvc?view=revision&revision=1387589 LUCENE-4406 , LUCENE-4407 : Upgrade to randomizedtesting 2.0.1. Workaround for broken test output XMLs due to non-XML text unicode chars in strings. Added printing of failed tests at the end of a test run (Dawid Weiss)
        Hide
        Uwe Schindler added a comment -

        Closed after release.

        Show
        Uwe Schindler added a comment - Closed after release.

          People

          • Assignee:
            Dawid Weiss
            Reporter:
            Grant Ingersoll
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development