[SUREFIRE-1336] Sure-Fire test run totals don't match TestNG report on failure.

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.19.1
• Component/s: TestNG support
• Environment: Oracle JDK 1.8.0_121, Maven 3.3.9, TestNG 6.10
• Assignee: Unassigned
• Reporter: Jeff Vincent
• Created: 21/Feb/17 20:30
• Updated: 21/Feb/17 20:30

Description

One of my TestNG test classes has 2 multi-threaded test methods that use DataProviders to feed in unique parameter values. In total, I have a test run that runs a total of 530,000 unique test instances between the two test methods.

In all test output files (XML, HTML, etc.), the test totals are correct (530,000 total, 22 failed, 529,978 passed.) The first test method runs 10,000 times, the 2nd test method runs 520,000 times. If all 520,000 pass, it reports 520,000 test cases succeed, but if any one of the iterations fail (or even multiple failures), then it only reports the single test method as a failure which is not accurate.

It seems that it isn't counting failures the same as it counts successes.