[BUILDR-301] TestNG doesn't report failure if more than one test fails

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.4
• Fix Version/s: 1.3.5
• Component/s: Test frameworks
• Environment: Tested on Linux (Fedora, Debian/Ubuntu)
• Assignee: Alex Boisvert
• Reporter: Martin Grotzke
• Created: 10/Aug/09 23:08
• Updated: 22/Jun/11 10:10
• Resolved: 17/Sep/09 23:48

Description

This issue is related to the report on the mailing list:
http://n2.nabble.com/Build-completes-successfully-on-configuration-failures-and-skipped-tests-td3332816.html#a3332816

I reproduced what we encountered in a simple project that has these test classes

• a BaseTest with a beforMethod that fails and a test method (that is skipped)
• a Test1 that extends BaseTest, with a test method that is also skipped.

This is the output of a `buildr clean test':
[grotzke@mescalin testmodule]$ buildr clean test
(in /home/grotzke/proj/buildr-sandbox, development)
Cleaning buildr-sandbox:testmodule
Testing buildr-sandbox:testmodule
Compiling buildr-sandbox:testmodule into /home/grotzke/proj/buildr-sandbox/testmodule/target/classes
Compiling buildr-sandbox:testmodule:test into /home/grotzke/proj/buildr-sandbox/testmodule/target/test/classes
Running tests in buildr-sandbox:testmodule
Running TestNG in buildr-sandbox:testmodule
[Parser] Running:
buildr-sandbox-testmodule

FAILED CONFIGURATION: @BeforeMethod beforeBaseTestMethod
java.lang.RuntimeException: s.th. failed here...
at de.javakaffee.buildr.sandbox.testmodule.BaseTest.beforeBaseTestMethod(BaseTest.scala:10)
... Removed 24 stack frames
SKIPPED CONFIGURATION: @BeforeMethod beforeBaseTestMethod
SKIPPED CONFIGURATION: @BeforeMethod beforeBaseTestMethod
SKIPPED: testBase
SKIPPED: testClass1
SKIPPED: testBase

===============================================
Command line test
Tests run: 3, Failures: 0, Skips: 3
Configuration Failures: 1, Skips: 2
===============================================

===============================================
buildr-sandbox-testmodule
Total tests run: 3, Failures: 0, Skips: 3
Configuration Failures: 1, Skips: 2
===============================================

Completed in 4.305s

The project can be found here:
http://github.com/magro/buildr-sandbox/tree/master