[SUREFIRE-1228] rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.19.1
• Component/s: TestNG support
• Assignee: Tibor Digana
• Reporter: Cam Morris
• Created: 10/Feb/16 15:14
• Updated: 07/Jun/20 22:27
• Resolved: 19/Sep/16 23:09

Description

Surefire confuses tests run with a data-provider with rerun tests so only one of the data-provided tests need to pass.

Steps to reproduce:

1. Create a testNG test with a data provider that passes with some of the data and fails with others
   TestNG example

     @Test(dataProvider = "succeed")
     public void fail(boolean succeed) {
   
       if (!succeed) {
         Assert.fail("nope");
       }
     }
   
     @DataProvider(name="succeed")
     public Object[][] arrayIndex() {
       return new Object[][]{ {true}, {false}};
     }
   

2. Configure surefire to rerun failed tests
   pom.xml example

   <?xml version="1.0" encoding="UTF-8"?>
   <project xmlns="http://maven.apache.org/POM/4.0.0"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
       <modelVersion>4.0.0</modelVersion>
   
       <groupId>cam.test</groupId>
       <artifactId>testng-retry</artifactId>
       <version>1.0-SNAPSHOT</version>
   
       <dependencies>
           <dependency>
               <groupId>org.testng</groupId>
               <artifactId>testng</artifactId>
               <version>6.9.10</version>
               <scope>test</scope>
           </dependency>
       </dependencies>
   
       <build>
   
           <plugins>
               <plugin>
                   <groupId>org.apache.maven.plugins</groupId>
                   <artifactId>maven-surefire-plugin</artifactId>
                   <version>2.19.1</version>
                   <configuration>
                     <rerunFailingTestsCount>1</rerunFailingTestsCount>
                   </configuration>
               </plugin>
           </plugins>
       </build>
   </project>
   

• Expected results: "Tests run: 2, Failures: 1..." The tests should not pass.
• Actual results: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1" The rerun confuses test runs with different data for retry attempts. Just one of the data provider tests needs to succeed for the test to be deemed a success.

  mvn test
  ...
  
  -------------------------------------------------------
   T E S T S
  -------------------------------------------------------
  Running TestRandomFail
  Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< FAILURE! - in TestRandomFail
  fail(TestRandomFail)  Time elapsed: 0.007 sec  <<< FAILURE!
  java.lang.AssertionError: nope
          at TestRandomFail.fail(TestRandomFail.java:14)
  
  
  Results :
  
  Flaked tests: 
  TestRandomFail.fail(TestRandomFail)
    Run 1: PASS
    Run 2: TestRandomFail.fail:14 nope
  
  
  Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
  
  [INFO] ------------------------------------------------------------------------
  [INFO] BUILD SUCCESS
  [INFO] ------------------------------------------------------------------------
  [INFO] Total time: 2.251 s
  [INFO] Finished at: 2016-02-10T07:39:21-07:00
  [INFO] Final Memory: 9M/244M
  [INFO] ------------------------------------------------------------------------