Run a JUnit 4 test marked with the @Ignore annotation. The test will not appear at all in the XML output. (I'll attach a sample project.)
Created attachment 21190 [details] sample project
Created attachment 21191 [details] resulting test output
I file this bug mostly because, as far as I know, the Ant team "owns" the format of JUnit XML that's consumed by various other tools (including Maven, Eclipse, CruiseControl, etc.) I don't know what the "expected" XML should look like (though it's pretty obvious that what we've currently got is wrong). Maybe something like this? <testsuite total="1" ignored="1"> <testcase name="testIgnorable> <ignore message="reason"/> </testcase> </testsuite> TestNG does this: <testsuite total="1" skipped="1"> <testcase name="testIgnorable> <skipped message="reason"/> </testcase> </testsuite>
Dan, as discussed on testng-user, this is because the current xml+report is junit3.x era. I am willing to extend it to support junit4, testng, antunit, smartfrog-xunit tests, etc, as long as we can do this without breaking : CruiseControl, Hudson, et al. Let's start by creating a wiki entry and listing -tools/products that generate the report -tools/products that process the XML it generates -features wanted -test plan Can I also point you at http://smartfrog.org/presentations/distributed_testing_with_smartfrog_slides.pdf and http://smartfrog.org/autolinks/googleLTAC06.htm where I show a prototype XHTML format that can be streamed out and is readable before the tests have finished. Moving to something like that would break everything, so I'm not going to propose it. But I would like to enable better streaming, by pushing all new summary information out as elements and not attributes on root nodes. I'd like something such as <testsuite total="1"> <metadata> <description>Tests of OS-specific things</description> <xml> <rdf xmlns:rdf="http://w3c.org/rdf"> <!-- random triples here --> </rdf> </xml> </metadata> <testcase name="testIgnorable"> <metadata> <description>A test that only works on Win98</description> <links> <uri>http://svn.example.org/ex/4</uri> </links> <tags><tag>win98</tag><tag>windows</tag><tag>trouble</tag></tags> <issues> <id>JIRA-117</id> <id>JIRA-113</id> </issues> </metadata> <skipped> wrong OS </skipped> </testcase> <summary> <total>1</total> <skipped>1</skipped> </summary> </testsuite> 1. new summary data goes in at the end. If you want to handle skipping, embrace the new format 2. extra stuff goes into a metadata corner. This is all static with every test. I'd like everything needed to create interesting Atom feeds.
Steve has created a wiki article here: http://wiki.apache.org/ant/Proposals/EnhancedTestReports
May not be straightforward to fix in the current <junit> task, which generally assumes JUnit 3.x and only handles 4.x tests using the compatibility wrapper. Real support for 4.x should I think be a separate task.
We are planning to migrate all our test cases to Junit4. Is this issue resolved yet? If not, are there any timelines on when will this feature be available. I would appreciate if I could get an update on this or if there is a workaround.
I'm not aware of anybody working on a JUnit4 specific Ant task, for most people our JUnit3 task seems to work well enough. So no, it is not fixed and can't be fixed easily since Ant's approach to JUnit4 makes the @ignore-d tests invisible to Ant. And it doesn't seem to be high on anybody's priority list either.
It's a pity that it's not high a in the priority, although the wiki proposal mentioned 1.8 as a possible target. Also, it seems that this tickets is not simply to support the @Ignore but to allow full support to junit-4 in ant. Hence, suggesting that in the updated <junit> task, a 'run-with' element would be welcome, allowing to force the junit-4 @RunWith in tests.
I have it working in R (svUnit, integrates nicely with Jenkins). This unit test suite manages to report a test as ignored by adding a further totally empty `skipped` element in the `testcase` element. http://stackoverflow.com/questions/3787886/producing-a-correct-and-complete-xml-unit-testing-report-for-hudson
Hi There - is there any update on this bug? Thanks, Harry.
Created attachment 30011 [details] Proposed Skipped Test Reporting Changes
I've added a patch with changes to report Ignored test (those marked with the @Ignore annotation) and skipped tests (those where Assume tests failed). The reporters bundled with the junit task have been modified to report these tests as skipped, and an additional interfaces has been added to allow users to add these events to their existing TaskListeners. The default XML report of changes contains the follow additional details with my patch: 1. An additional attribute on the root testsuite element named 'skipped' that contains the number of ignored and skipped tests 2. The test attribute on the root testsuite element now counts all tests (both run and ignored), not just the ones that have been run 3. All tests are listed as testcase elements, not just the ones that haven't been ignored 4. Any test that has been ignored or skipped has a child 'skipped' element that, if a message was given on the @Ignore annotation or a message given for an Assume failing, includes a message attribute containing that message This bring the XML document in-line with the XMl reports from other runners (e.g. Maven Surefire) and other test suites (e.g. TestNG) Patch attached as 'Proposed Skipped Test Reporting Changes'. Any comments, ideas, suggestions or issues are welcome.
Created attachment 30012 [details] Updated Patch Updated patch: 1. Separate JUnit4 code out from formatters so users can continue to run using JUnit3 2. Support retrieving @Ignore messages from JUnit4 versions prior to version 4.4 This new patch has been tested against both JUnit3 and JUnit4 with no issues.
Created attachment 30013 [details] Example Patched XML Output Example XML output with explicitly ignored test (using @Ignore) and skipped tests (failed Assume), as well as errors, failures and successes.
Created attachment 30014 [details] Updated patch Patch with test case for checking XML format re-instated
Created attachment 30015 [details] Updated Patch +test -duplication Patch with test case re-instated and duplicate code removed. The previous patch introduced a CustomJUnit4TestAdapterCache which provided the notifier for Ignored and Skipped events in JUnit4. This new class mostly duplicated the existing private 'Cache' class within JUnit4TestMethodAdapter, although added the methods needed for Ignored and Skipped events. This latest patch removes the private class and changes references to it to the new public class so as to prevent code duplication.
Fixed. Thanks Michael for the big contribution.
Created attachment 30092 [details] Ant junit xsl style sheets modified to show skipped tests Attached modified junit-frames.xsl and junit-noframes.xsl to show skipped count. In 'junitreport' task, use the 'styledir' attribute of the 'report' element to specify the directory where to find the style sheets.
Gilles: I've already raised a further issue for this with an associated patch - Bug 54670. I've not checked your patch but assume it's similar to what I've done, if not then feel free to attach it to the new issue.