Uploaded image for project: 'Maven Surefire'
  1. Maven Surefire
  2. SUREFIRE-1691

surefire-junit47 reports results against incorrect test



    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.0.0-M3
    • None
    • Junit 4.x support
    • None


      This is a follow up on:

      The latest reproducer can be found here:

      In short Cucumber has a hierarchy of tests that looks at follows:

      Class annotated with @RunWith(Cucumber.class)
       |- Feature File 1
       |  |- Scenario 1a
       |  |- Scenario 1b
       |- Feature File 2
       |  |- Scenario 2a
       |  |- Scenario 2b

      Surefire tries to group output in tests sets, either per class or per test suite. As Cucumber currently only emits test started/finished events for Scenarios sure fire can not group scenarios into a test set. They are instead grouped with the previous test set. However even after adding test started/finished events for features, sure fire groups all results in a single test set.

      I believe this is caused by an error in NonConcurrentRunListener.describesNewTestSet.

          private boolean describesNewTestSet( Description description )
              if ( currentTestSetDescription != null )
                  if ( null != description.getTestClass() )
                      return !description.getTestClass().equals( currentTestSetDescription.getTestClass() );
                  else if ( description.isSuite() )
                      return description.getChildren().equals( currentTestSetDescription.getChildren() );
                  return false;
              return true;

      The value of description.getChildren().equals( currentTestSetDescription.getChildren() ); should be negated so so different test suits are properly recognized as test sets.

      Fixing this locally works and can be reproduced by:

      1. Negating the line and building sure fire locally
      2. Building https://github.com/cucumber/cucumber-jvm/pull/1765 with `mvn clean install -DskipTests`
      3. Running `mvn clean test -Pjunit47` on https://github.com/mpkorstanje/surefire-test

      It does however does result in another unwanted result: A feature file with two scenarios is now reported as having 3 results. One for each scenario and one for the feature.

      Test set: Some Feature 1
      Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 s - in Some_Feature_1




            Unassigned Unassigned
            mpkorstanje M.P. Korstanje
            0 Vote for this issue
            2 Start watching this issue