Uploaded image for project: 'Maven Surefire'
  1. Maven Surefire
  2. SUREFIRE-2276

JUnit5's TestTemplate failures treated as flakes with retries

    XMLWordPrintableJSON

Details

    Description

      G'day from Atlassian,

      TLDR: When surefire.rerunFailingTestsCount is set to 1 or higher, then test template test failures are classified as flakes rather than failures.

      Failing test case demonstrating the problem at https://github.com/apache/maven-surefire/compare/master...atlassian-forks:maven-surefire:master

      JUnit5 has a feature called Test Templates (docs) which is very similar to the well known parametrised tests. The main difference is that in a test template, we can not only provide parameters for tests, but the entire context, including lifecycle methods (@Before..., @After... etc.).

      When one of the test method runs fails, Surefire correctly summarises the failure and exits with a failure. E.g. if we provided the test template method with two contexts, of which one reliably fails and one succeeds, then Surefire would, just like JUnit, show that it had run two tests, of which one failed.

      There is a problem with surefire.rerunFailingTestsCount though. The setting correctly causes Surefire to invoke retries on the right tests, but the final summary of failures is wrong. With one working and one consistently failing test context, the failing one would be re-run, but the final summary would say that we had only ONE test and that it was flaky. This causes Surefire to exit with a success code, even though tests did fail.

      Interestingly, the problem does NOT surface when a parametrised test fails consistently, even though parametrised tests use test templates internally. There seems to be very specific logic for handling parametrised tests, which does not cover test templates around here: https://github.com/apache/maven-surefire/blob/d7f4dbb7cc99a08a911066afbab3e01273dbbdc0/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java#L325.

       

      The test case in https://github.com/apache/maven-surefire/compare/master...atlassian-forks:maven-surefire:master shows how adding the retries changes the test overall outcome and the summary stats.
      Expected: 2 run, 1 failure. Overall: fail
      Actual: 1 run, 1 flake. Overall: success

      I also added a test class for parametrised tests, where one param makes the test consistently fail. The test class is commented out in the IT, but might help with debugging.

       

      We are currently investigating further and trying to get a patch ready.

      EDIT: We have a patch ready for merging at https://github.com/apache/maven-surefire/pull/788

      Attachments

        Issue Links

          Activity

            People

              michael-o Michael Osipov
              huuugo Hugo G
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: