Uploaded image for project: 'Maven Surefire'
  1. Maven Surefire
  2. SUREFIRE-1691

surefire-junit47 reports results against incorrect test

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.0.0-M3
    • 3.0
    • Junit 4.x support
    • None

    Description

      This is a follow up on:

      The latest reproducer can be found here:

      In short Cucumber has a hierarchy of tests that looks at follows:

      Class annotated with @RunWith(Cucumber.class)
       |- Feature File 1
       |  |- Scenario 1a
       |  |- Scenario 1b
       |- Feature File 2
       |  |- Scenario 2a
       |  |- Scenario 2b
      

      Surefire tries to group output in tests sets, either per class or per test suite. As Cucumber currently only emits test started/finished events for Scenarios sure fire can not group scenarios into a test set. They are instead grouped with the previous test set. However even after adding test started/finished events for features, sure fire groups all results in a single test set.

      I believe this is caused by an error in NonConcurrentRunListener.describesNewTestSet.

          private boolean describesNewTestSet( Description description )
          {
              if ( currentTestSetDescription != null )
              {
                  if ( null != description.getTestClass() )
                  {
                      return !description.getTestClass().equals( currentTestSetDescription.getTestClass() );
                  }
                  else if ( description.isSuite() )
                  {
                      return description.getChildren().equals( currentTestSetDescription.getChildren() );
                  }
      
                  return false;
              }
      
              return true;
          }
      

      The value of description.getChildren().equals( currentTestSetDescription.getChildren() ); should be negated so so different test suits are properly recognized as test sets.

      Fixing this locally works and can be reproduced by:

      1. Negating the line and building sure fire locally
      2. Building https://github.com/cucumber/cucumber-jvm/pull/1765 with `mvn clean install -DskipTests`
      3. Running `mvn clean test -Pjunit47` on https://github.com/mpkorstanje/surefire-test

      It does however does result in another unwanted result: A feature file with two scenarios is now reported as having 3 results. One for each scenario and one for the feature.

      -------------------------------------------------------------------------------
      Test set: Some Feature 1
      -------------------------------------------------------------------------------
      Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 s - in Some_Feature_1
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            mpkorstanje M.P. Korstanje
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: