Uploaded image for project: 'Maven Surefire'
  1. Maven Surefire
  2. SUREFIRE-1470

Unexpected report result when using rerunFailingTestsCount with groups filter in Surefire

    Details

      Description

      When the following plugin configuration is used:

      <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>${surefire.version}</version>
          <configuration>
              <includes>
                  <include>**/*.java</include>
              </includes>
              <groups>TestGroup</groups>
              <rerunFailingTestsCount>1</rerunFailingTestsCount>
          </configuration>
      </plugin>

      And there is a test that fails first time it's run, the resultant txt report is unexpected and only contains the result of last rerun.

      Suppose we have the following test class:

      @Category(TestGroup.class)
      public class CustomTest {
      
          @Test
          public void alwaysPassingTest() {
              Assert.assertEquals(0, 0);
          }
      
          private static volatile int x = 0;
      
          @Test
          public synchronized void secondAttemptPassingTest() throws InterruptedException {
              Thread.sleep(5000);
              if (x++ == 0) {
                  Assert.fail();
              }
          }
      
      }
      

      The resultant xml report is all right, but the txt one is

      -------------------------------------------------------------------------------
      Test set: CustomTest
      -------------------------------------------------------------------------------
      Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063 s - in CustomTest
      

      Which is basically incorrect. The txt file is regenerated several times during the test run.

      The issue is present when groups filter (either exclusion or inclusion) is present. When there is no group filter and only rerunFailingTestsCount is configured, then the resultant report is the following one:

      -------------------------------------------------------------------------------
      Test set: CustomTest
      -------------------------------------------------------------------------------
      Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.065 s <<< FAILURE! - in CustomTest
      secondAttemptPassingTest(CustomTest)  Time elapsed: 0.009 s  <<< FAILURE!
      java.lang.AssertionError
      	at CustomTest.secondAttemptPassingTest(CustomTest.java:18)
      

      When no group filter is configured, Junit4Provider instead of JunitCore provider is picked up and they seem to handle those cases differently.

      The situation is worsened by the fact that TeamCity is not able to parse the results correctly when the groups are configured along with rerunFailingTestsCount and it marks the tests that passed after rerun as failed, although the overall Maven result is SUCCESS in both cases.

      The simple Maven project to reproduce the issue is attached.

        Attachments

        1. test.zip
          5 kB
          Alexey Subach

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              alexey.subach Alexey Subach
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: