[SUREFIRE-1691] surefire-junit47 reports results against incorrect test

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-M3
• Component/s: Junit 4.x support
• Assignee: Unassigned
• Reporter: M.P. Korstanje
• Created: 01/Sep/19 12:36
• Updated: 30/May/23 18:11

Description

This is a follow up on:

• https://issues.apache.org/jira/browse/SUREFIRE-1156
• https://github.com/cucumber/cucumber-jvm/issues/865

The latest reproducer can be found here:

• https://github.com/mpkorstanje/surefire-test/tree/07d0ee98ea424b38a3c548981a875dde1ef4f2c2

In short Cucumber has a hierarchy of tests that looks at follows:

Class annotated with @RunWith(Cucumber.class)
 |- Feature File 1
 |  |- Scenario 1a
 |  |- Scenario 1b
 |- Feature File 2
 |  |- Scenario 2a
 |  |- Scenario 2b

Surefire tries to group output in tests sets, either per class or per test suite. As Cucumber currently only emits test started/finished events for Scenarios sure fire can not group scenarios into a test set. They are instead grouped with the previous test set. However even after adding test started/finished events for features, sure fire groups all results in a single test set.

I believe this is caused by an error in NonConcurrentRunListener.describesNewTestSet.

    private boolean describesNewTestSet( Description description )
    {
        if ( currentTestSetDescription != null )
        {
            if ( null != description.getTestClass() )
            {
                return !description.getTestClass().equals( currentTestSetDescription.getTestClass() );
            }
            else if ( description.isSuite() )
            {
                return description.getChildren().equals( currentTestSetDescription.getChildren() );
            }

            return false;
        }

        return true;
    }

The value of description.getChildren().equals( currentTestSetDescription.getChildren() ); should be negated so so different test suits are properly recognized as test sets.

Fixing this locally works and can be reproduced by:

1. Negating the line and building sure fire locally
2. Building https://github.com/cucumber/cucumber-jvm/pull/1765 with `mvn clean install -DskipTests`
3. Running `mvn clean test -Pjunit47` on https://github.com/mpkorstanje/surefire-test

It does however does result in another unwanted result: A feature file with two scenarios is now reported as having 3 results. One for each scenario and one for the feature.

-------------------------------------------------------------------------------
Test set: Some Feature 1
-------------------------------------------------------------------------------
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 s - in Some_Feature_1