Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: test
    • Labels:
      None

      Description

      Amongst other things, JUnit 4 has better support for class-wide set up and tear down (via @BeforeClass and @AfterClass annotations), and more flexible assertions (http://junit.sourceforge.net/doc/ReleaseNotes4.4.html). It would be nice to be able to take advantage of these features in tests we write.

      JUnit 4 can run tests written for JUnit 3.8.1 without any changes.

        Issue Links

          Activity

          Hide
          Nigel Daley added a comment -

          Or should we consider TestNG? It would be nice to be able to categorize tests into 1 or more categories (smoke, nightly, dfs, mapred, etc). Not sure JUnit 4 can do this, but TestNG can.

          On the other hand, TestNG seems to lack the nice console output controls that JUnit provides.

          Show
          Nigel Daley added a comment - Or should we consider TestNG? It would be nice to be able to categorize tests into 1 or more categories (smoke, nightly, dfs, mapred, etc). Not sure JUnit 4 can do this, but TestNG can. On the other hand, TestNG seems to lack the nice console output controls that JUnit provides.
          Hide
          Hemanth Yamijala added a comment -

          It would be nice to be able to categorize tests into 1 or more categories (smoke, nightly, dfs, mapred, etc).

          +1. We have been seeing bug reports of tests taking a long time to run. These have all been correct reports so far because the tests were inefficient and hence could be fixed. But I suspect there might be legitimate cases in future where some tests will actually require a longer time to run and we want them as part of the automated test suite. For eg we are trying to enhance capacity scheduler test cases to start using the MiniMRCluster. The objective of these tests is to test the capacity scheduler along with the JobTracker, above and beyond unit tests for the scheduler itself. Clearly these need not be run for every patch submitted to Hudson, but maybe once every day or so. That way, you could add long running tests without incurring overhead per patch.

          Any new test framework that would allow this flexibility would be great to have !

          Show
          Hemanth Yamijala added a comment - It would be nice to be able to categorize tests into 1 or more categories (smoke, nightly, dfs, mapred, etc). +1. We have been seeing bug reports of tests taking a long time to run. These have all been correct reports so far because the tests were inefficient and hence could be fixed. But I suspect there might be legitimate cases in future where some tests will actually require a longer time to run and we want them as part of the automated test suite. For eg we are trying to enhance capacity scheduler test cases to start using the MiniMRCluster. The objective of these tests is to test the capacity scheduler along with the JobTracker, above and beyond unit tests for the scheduler itself. Clearly these need not be run for every patch submitted to Hudson, but maybe once every day or so. That way, you could add long running tests without incurring overhead per patch. Any new test framework that would allow this flexibility would be great to have !
          Hide
          Tom White added a comment -

          Categories would be nice to have. The documentation for JUnit 4.4 at http://junit.sourceforge.net/doc/ReleaseNotes4.4.html mentions categories, and says

          This feature will very likely be improved before the final release of JUnit 4.5

          However I can't find a Category annotation in 4.5, so perhaps it hasn't been added yet. There is a JUnit Extension project at http://junitext.sourceforge.net/ (Apache-licensed) which does define a Category annotation for this purpose. (I haven't tried it.)

          Here is a comparison of TestNG and JUnit 4: http://xplanner.codehaus.org/Migration+to+JUnit4+or+TestNG. TestNG requires source migration of tests, whereas JUnit 4 is just a change of jar.

          BTW I wrote a blog entry with some other ideas for improving testing at http://www.cloudera.com/blog/2008/12/16/testing-hadoop/

          Show
          Tom White added a comment - Categories would be nice to have. The documentation for JUnit 4.4 at http://junit.sourceforge.net/doc/ReleaseNotes4.4.html mentions categories, and says This feature will very likely be improved before the final release of JUnit 4.5 However I can't find a Category annotation in 4.5, so perhaps it hasn't been added yet. There is a JUnit Extension project at http://junitext.sourceforge.net/ (Apache-licensed) which does define a Category annotation for this purpose. (I haven't tried it.) Here is a comparison of TestNG and JUnit 4: http://xplanner.codehaus.org/Migration+to+JUnit4+or+TestNG . TestNG requires source migration of tests, whereas JUnit 4 is just a change of jar. BTW I wrote a blog entry with some other ideas for improving testing at http://www.cloudera.com/blog/2008/12/16/testing-hadoop/
          Hide
          Jim Kellerman added a comment -

          +1 for JUnit4

          Show
          Jim Kellerman added a comment - +1 for JUnit4
          Hide
          Steve Loughran added a comment -

          I've played with both of these. In some ways I prefer testng to junit4. Junit4 is a better drop-in for the ant tasks, though reporting through <junit> is a bit degraded. Someone and it would probalby end up being me would have to improve things there by having a separate antlib for junit4 that is java5+ only.

          TestNG has a very complex interdependency system, all configured from XML. Its powerful, but can get a bit confusing, and some people are against it because its ordering stops tests being independent. I think that's a unit test purist opinion. What they do have is some data-driven testing where a CSV file drives the same method with different parameters; this could be interesting.

          Both of them treat functional testing of a system cluster as a bit of an afterthought, as do the standard XML->HTML reports. I've done lots of work in testing smartforg by using it as a functional test runner itself: deploying tests and collecting the results. That is also something hadoop's functional tests could do, feeding the results back into the DFS for later analysis.

          Show
          Steve Loughran added a comment - I've played with both of these. In some ways I prefer testng to junit4. Junit4 is a better drop-in for the ant tasks, though reporting through <junit> is a bit degraded. Someone and it would probalby end up being me would have to improve things there by having a separate antlib for junit4 that is java5+ only. TestNG has a very complex interdependency system, all configured from XML. Its powerful, but can get a bit confusing, and some people are against it because its ordering stops tests being independent. I think that's a unit test purist opinion. What they do have is some data-driven testing where a CSV file drives the same method with different parameters; this could be interesting. Both of them treat functional testing of a system cluster as a bit of an afterthought, as do the standard XML->HTML reports. I've done lots of work in testing smartforg by using it as a functional test runner itself: deploying tests and collecting the results. That is also something hadoop's functional tests could do, feeding the results back into the DFS for later analysis.
          Hide
          Steve Loughran added a comment -

          Some more thoughts here:
          -regardless of what framework gets used, the report XML needs to be redone. This is something I've discussed in Ant before, and a <junit4> task would be an opportunity to go forwards with a newer format. Even if still XML, it shouldnt buffer everything in memory until the the end of the run just to add some statistics as a attributes on the root node.

          -we'd have to work with the maven2 and test ng teams, as well as the various CI tools to see that it met their needs too. Or we could do a hadoop-specific one that pushed results and log data straight into a DFS file.

          Show
          Steve Loughran added a comment - Some more thoughts here: -regardless of what framework gets used, the report XML needs to be redone. This is something I've discussed in Ant before, and a <junit4> task would be an opportunity to go forwards with a newer format. Even if still XML, it shouldnt buffer everything in memory until the the end of the run just to add some statistics as a attributes on the root node. -we'd have to work with the maven2 and test ng teams, as well as the various CI tools to see that it met their needs too. Or we could do a hadoop-specific one that pushed results and log data straight into a DFS file.
          Hide
          Tom White added a comment -

          Avro (AVRO-26) and ZooKeeper (ZOOKEEPER-397) have moved (or are in the process of moving) to TestNG.

          Show
          Tom White added a comment - Avro ( AVRO-26 ) and ZooKeeper ( ZOOKEEPER-397 ) have moved (or are in the process of moving) to TestNG.
          Hide
          Philip Zeyliger added a comment -

          Would we consider the transition world OK where we had tests in both frameworks for a while? I believe the TestNG runner can run JUnit tests, so we could probably start adding new tests in TestNG, and transition old ones as they come up.

          Show
          Philip Zeyliger added a comment - Would we consider the transition world OK where we had tests in both frameworks for a while? I believe the TestNG runner can run JUnit tests, so we could probably start adding new tests in TestNG, and transition old ones as they come up.
          Hide
          Steve Loughran added a comment -

          testng runs junit3 tests, not junit4. So it will run the existing tests, but not if you try some junit4 stuff and then back off that route.

          Show
          Steve Loughran added a comment - testng runs junit3 tests, not junit4. So it will run the existing tests, but not if you try some junit4 stuff and then back off that route.
          Hide
          Philip Zeyliger added a comment -

          I believe one of our goals is to be able to "label" tests as "smoke", "fast", "slow", etc. One way to approach this, even with JUnit3, is to use Java annotations, and then query for them with something like http://developer.android.com/reference/android/test/suitebuilder/TestSuiteBuilder.html . We could easily cook up something similar.

          Show
          Philip Zeyliger added a comment - I believe one of our goals is to be able to "label" tests as "smoke", "fast", "slow", etc. One way to approach this, even with JUnit3, is to use Java annotations, and then query for them with something like http://developer.android.com/reference/android/test/suitebuilder/TestSuiteBuilder.html . We could easily cook up something similar.
          Hide
          Konstantin Boudnik added a comment -

          Or we can start using testng to run JUnit tests natively and migrate them to
          'native' TestNG format eventually. There's a couple of benefits which JUnit
          doesn't have. E.g.

          • parallel tests execution
          • sameVM execution (without forking)
          • absence of mandatory TestCase's inheritance which limits the options of
            test developers.

          With regards,
          Cos

          Show
          Konstantin Boudnik added a comment - Or we can start using testng to run JUnit tests natively and migrate them to 'native' TestNG format eventually. There's a couple of benefits which JUnit doesn't have. E.g. parallel tests execution sameVM execution (without forking) absence of mandatory TestCase's inheritance which limits the options of test developers. With regards, Cos
          Hide
          Adrian Woodhead added a comment -

          With regards to the comment above:

          "parallel tests execution" - while this could be useful for speeding tests up, I see potential problems here if different tests access a shared resource (database, file system etc.)
          "same VM execution" - JUnit has this. When you run it from Ant you can specify whether it should use separate VM to Ant and also whether it should use a separate VM per test or not.
          "absence of mandatory TestCase inheritance" - JUnit4 has this.

          So I'm not convinced that these arguments are enough to tip the balance in favour of TestNG.

          My vote would go for JUnit4 in terms of keeping the transition easier and it being generally more familiar.

          Show
          Adrian Woodhead added a comment - With regards to the comment above: "parallel tests execution" - while this could be useful for speeding tests up, I see potential problems here if different tests access a shared resource (database, file system etc.) "same VM execution" - JUnit has this. When you run it from Ant you can specify whether it should use separate VM to Ant and also whether it should use a separate VM per test or not. "absence of mandatory TestCase inheritance" - JUnit4 has this. So I'm not convinced that these arguments are enough to tip the balance in favour of TestNG. My vote would go for JUnit4 in terms of keeping the transition easier and it being generally more familiar.
          Hide
          Steve Loughran added a comment -

          -you can run junit4 under <junit>, but the reporting doesn't improve; you don't get a log of skipped tests, for example. One stevel has been nominated to fix this, but I'd need help with the XSL transforms. Or somehow get rid of that phase altogether.

          -for any improved Ant tasks, I'd expect <antunit> to do the testing, it generates the same xml output as <junit>

          Show
          Steve Loughran added a comment - -you can run junit4 under <junit>, but the reporting doesn't improve; you don't get a log of skipped tests, for example. One stevel has been nominated to fix this, but I'd need help with the XSL transforms. Or somehow get rid of that phase altogether. -for any improved Ant tasks, I'd expect <antunit> to do the testing, it generates the same xml output as <junit>
          Hide
          Konstantin Boudnik added a comment -

          Adding on the same topic. I've investigated JUnit4 more closely and while this new version is getting much more advanced compare to the old one, it is still lacking a number of features provided by TestNG. It seems that even grouping is somehow supported now, alas TestNG feature is a way more complex and flexible.

          Here I've found pretty good comparison between the latest versions of the harnesses:
          http://lijinjoseji.wordpress.com/2008/02/29/testng-56-and-junit-44-which-framework-you-will-choose-for-unit-testing/

          Obviously, the decision about such switching from one harness to another has to be made based on an actual set of feature the project uses or will be using in the feature.

          From a personal experience, I could say that such a conversion might take quite an effort if a project's tests have a heavy JUnit legacy such careless resources allocation/release and similar things.

          Show
          Konstantin Boudnik added a comment - Adding on the same topic. I've investigated JUnit4 more closely and while this new version is getting much more advanced compare to the old one, it is still lacking a number of features provided by TestNG. It seems that even grouping is somehow supported now, alas TestNG feature is a way more complex and flexible. Here I've found pretty good comparison between the latest versions of the harnesses: http://lijinjoseji.wordpress.com/2008/02/29/testng-56-and-junit-44-which-framework-you-will-choose-for-unit-testing/ Obviously, the decision about such switching from one harness to another has to be made based on an actual set of feature the project uses or will be using in the feature. From a personal experience, I could say that such a conversion might take quite an effort if a project's tests have a heavy JUnit legacy such careless resources allocation/release and similar things.
          Hide
          Steve Loughran added a comment -

          a lot of the tests are a bit weak on releasing the miniMR clusters on failure. As long as everything works, nobody notices, but when a test fails, the clusters hang around.
          It's that cluster setup/teardown that causes a lot of the tests to be slow -if teardown could be asynchronous, maybe even with queued creation of MiniMRClusters, things could be faster.

          Show
          Steve Loughran added a comment - a lot of the tests are a bit weak on releasing the miniMR clusters on failure. As long as everything works, nobody notices, but when a test fails, the clusters hang around. It's that cluster setup/teardown that causes a lot of the tests to be slow -if teardown could be asynchronous, maybe even with queued creation of MiniMRClusters, things could be faster.
          Hide
          Steve Loughran added a comment -

          remember that some of the criticisms of junit4 are really of the junit test runner from Ant. we have the capability to fix that, and release that fix on a schedule which suits the Hadoop team.

          Show
          Steve Loughran added a comment - remember that some of the criticisms of junit4 are really of the junit test runner from Ant. we have the capability to fix that, and release that fix on a schedule which suits the Hadoop team.
          Hide
          Konstantin Boudnik added a comment -

          Right, there are issues with the runner from Ant. However, I can't possibly
          see how, say, test parametrization might be solved from this side?

          Show
          Konstantin Boudnik added a comment - Right, there are issues with the runner from Ant. However, I can't possibly see how, say, test parametrization might be solved from this side?
          Hide
          Jon Kean added a comment -

          I stumbled upon this request while looking for something related. My team at Palantir Technologies (www.palantirtech.com) encountered this exact situation, so I thought I'd add a quick comment or two about our experience.

          Parallel execution is supported by JUnit 4.6, using an experimental test running. Although the runner has a bugs and limitations, and is unusable in the 4.6 release. I recently submitted a patch to correct the problems (should make Junit 4.7, but for now my team uses the patched runner with JUnit 4.4). We've used the parallel runner with our continuous suite of ~8K tests, that runs 24/7 without drama. Many of our tests use slow RMI calls, so we saw a near linear increase in speed with our current build box using 16 test threads with roughly a 12x speedup in test time. The only caveat is the occasional uninformed developer writing unit tests not suitable for parallel execution (ie: a static resource used by many test classes), but that problem is usually quickly uncovered.

          JUnit does support parameter/data testing, using the Parameterized annotation. The feature is awesome for randomized fuzz testing, especially if you have lots of code implementing only a few interfaces.

          Test categorization is not built into JUnit 4, although JUnitExt supports it. If pulling in another library dependency is undesirable, you can always use multiple root test suites (ie: continuous, nightly, smoke, etc). It's simple and works, but every once in a while someone will check in a new test and forget to add it to a suite.

          JUnit 4 is literally a drop in solution (assuming one is testing with a Java 5 VM), while TestNG requires at least some change.

          Ultimately, we decided TestNG gave very little benefit over JUnit + extensions (dare I say no benefit?), and decided to stick with JUnit as it has better adoption and devs are already familiar with it.

          Show
          Jon Kean added a comment - I stumbled upon this request while looking for something related. My team at Palantir Technologies (www.palantirtech.com) encountered this exact situation, so I thought I'd add a quick comment or two about our experience. Parallel execution is supported by JUnit 4.6, using an experimental test running. Although the runner has a bugs and limitations, and is unusable in the 4.6 release. I recently submitted a patch to correct the problems (should make Junit 4.7, but for now my team uses the patched runner with JUnit 4.4). We've used the parallel runner with our continuous suite of ~8K tests, that runs 24/7 without drama. Many of our tests use slow RMI calls, so we saw a near linear increase in speed with our current build box using 16 test threads with roughly a 12x speedup in test time. The only caveat is the occasional uninformed developer writing unit tests not suitable for parallel execution (ie: a static resource used by many test classes), but that problem is usually quickly uncovered. JUnit does support parameter/data testing, using the Parameterized annotation. The feature is awesome for randomized fuzz testing, especially if you have lots of code implementing only a few interfaces. Test categorization is not built into JUnit 4, although JUnitExt supports it. If pulling in another library dependency is undesirable, you can always use multiple root test suites (ie: continuous, nightly, smoke, etc). It's simple and works, but every once in a while someone will check in a new test and forget to add it to a suite. JUnit 4 is literally a drop in solution (assuming one is testing with a Java 5 VM), while TestNG requires at least some change. Ultimately, we decided TestNG gave very little benefit over JUnit + extensions (dare I say no benefit?), and decided to stick with JUnit as it has better adoption and devs are already familiar with it.
          Hide
          Konstantin Boudnik added a comment -

          Here's a comment on JUnitExt remarks. It seems that JUnit Extension project supports the categorization of some kind. However, it seems that the semantic of their @Category is very different from our desire for tagging is. Here's the excerpt from JUnitExt tutorial:

          JUnitExt tutorial's excerpt

          How to use categories

          Tests can be categorized, to sort them for different puporses. JUnitExt provides a CategoryTextListerner, which prints out at end of test run all categories, with status of tests listed: May be Success, Ignored, Failed. You can plugin the category text listener using following code: (see also org.junitext.samples package)

          JUnitCore core = new JUnitCore();
          // use for categories special listener, give some statistics
          core.addListener(new CategoryTextListener(System.out));
          core.run(SimpleTest.class);

          The output when running thes tests will be:

          I.
          Time: 0

          OK (1 test)

          Category: equal tests
          Success testEquals(org.junitext.samples.SimpleTest)
          Category: math tests
          Ignored divideByZero(org.junitext.samples.SimpleTest)

          It might be possible to extend or develop new ClassRunner which will be capable of running only tests from a specified @Category, however it isn't provided by JUnitExt out of the box or the documentation fails no mention it.

          Show
          Konstantin Boudnik added a comment - Here's a comment on JUnitExt remarks. It seems that JUnit Extension project supports the categorization of some kind. However, it seems that the semantic of their @Category is very different from our desire for tagging is. Here's the excerpt from JUnitExt tutorial: JUnitExt tutorial's excerpt How to use categories Tests can be categorized, to sort them for different puporses. JUnitExt provides a CategoryTextListerner, which prints out at end of test run all categories, with status of tests listed: May be Success, Ignored, Failed. You can plugin the category text listener using following code: (see also org.junitext.samples package) JUnitCore core = new JUnitCore(); // use for categories special listener, give some statistics core.addListener(new CategoryTextListener(System.out)); core.run(SimpleTest.class); The output when running thes tests will be: I. Time: 0 OK (1 test) Category: equal tests Success testEquals(org.junitext.samples.SimpleTest) Category: math tests Ignored divideByZero(org.junitext.samples.SimpleTest) It might be possible to extend or develop new ClassRunner which will be capable of running only tests from a specified @Category, however it isn't provided by JUnitExt out of the box or the documentation fails no mention it.
          Hide
          Konstantin Boudnik added a comment -

          I've done some additional investigation on Spring-test framework and it seems that the tagging solution is already in place there. Here's an example:

          MyTests.java
          @RunWith(SpringJUnit4ClassRunner.class)
          @TestExecutionListeners( {})
          public class MyTests {
          	@BeforeClass
          	public static void setUp() {
          		System.out.println("test-group is set to: " + System.getProperty("test-group"));
          	}
          
          	@Test
          	@IfProfileValue(name = "test-group", values = { "fast", "quick" })
          	public void testOne() {
          		System.out.println("testOne(): Quick stuff");
          	}
          
          	@Test
          	@IfProfileValue(name = "test-group", value = "fast")
          	public void testTwo() {
          		System.out.println("testTwo(): Fast stuff");
          	}
          }
          

          then to run say 'fast' tests one just need to specify -Dtest-group=fast at run time.

          This solution requires a couple of jars to be pulled down from Maven repo, but other than that is it quite clear and easy to implement.

          I have found a couple of the issues with Spring-test framework which will affect this approach or at least might slightly postpone it. See http://jira.springframework.org/browse/SPR-5145 and but the fix is coming in the release 3.0 of Spring, which is around the corner.

          Other minor issues are: http://jira.springframework.org/browse/SPR-5902, http://jira.springframework.org/browse/SPR-5903 but these are rather improvements than bugs.

          Show
          Konstantin Boudnik added a comment - I've done some additional investigation on Spring-test framework and it seems that the tagging solution is already in place there. Here's an example: MyTests.java @RunWith(SpringJUnit4ClassRunner.class) @TestExecutionListeners( {}) public class MyTests { @BeforeClass public static void setUp() { System .out.println( "test-group is set to: " + System .getProperty( "test-group" )); } @Test @IfProfileValue(name = "test-group" , values = { "fast" , "quick" }) public void testOne() { System .out.println( "testOne(): Quick stuff" ); } @Test @IfProfileValue(name = "test-group" , value = "fast" ) public void testTwo() { System .out.println( "testTwo(): Fast stuff" ); } } then to run say 'fast' tests one just need to specify -Dtest-group=fast at run time. This solution requires a couple of jars to be pulled down from Maven repo, but other than that is it quite clear and easy to implement. I have found a couple of the issues with Spring-test framework which will affect this approach or at least might slightly postpone it. See http://jira.springframework.org/browse/SPR-5145 and but the fix is coming in the release 3.0 of Spring, which is around the corner. Other minor issues are: http://jira.springframework.org/browse/SPR-5902 , http://jira.springframework.org/browse/SPR-5903 but these are rather improvements than bugs.
          Hide
          Konstantin Boudnik added a comment -

          It seems that Common is using Junit-4.5 already. So, I'd suggest to close this JIRA as resolved.

          Show
          Konstantin Boudnik added a comment - It seems that Common is using Junit-4.5 already. So, I'd suggest to close this JIRA as resolved.
          Hide
          Alex Loddengaard added a comment -

          Done. Please reopen if appropriate. Thanks!

          Alex

          Show
          Alex Loddengaard added a comment - Done. Please reopen if appropriate. Thanks! Alex
          Hide
          Tom White added a comment -

          JUnit 4.8, which was released a few days ago (http://www.junit.org/node/581), adds support for categories: http://kentbeck.github.com/junit/doc/ReleaseNotes4.8.html

          Show
          Tom White added a comment - JUnit 4.8, which was released a few days ago ( http://www.junit.org/node/581 ), adds support for categories: http://kentbeck.github.com/junit/doc/ReleaseNotes4.8.html
          Hide
          Konstantin Boudnik added a comment -

          Well they kinda took SpringTest approach but I don't like the fact that one use classes as the category tags. Looks kinda ugly. I'd suggest to hold for a while until this implementation will become not preliminary but permanent.

          Show
          Konstantin Boudnik added a comment - Well they kinda took SpringTest approach but I don't like the fact that one use classes as the category tags. Looks kinda ugly. I'd suggest to hold for a while until this implementation will become not preliminary but permanent.

            People

            • Assignee:
              Alex Loddengaard
              Reporter:
              Tom White
            • Votes:
              4 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development