Buildr
  1. Buildr
  2. BUILDR-62

JUnit test filtering should allow custom filtering, and not test JUnit @Test annotation on class

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.3
    • Fix Version/s: 1.3
    • Component/s: Test frameworks
    • Labels:
      None

      Description

      If I understand correctly, in test_frameworks.rb

      http://svn.apache.org/repos/asf/incubator/buildr/trunk/lib/buildr/java/test_frameworks.rb

      JUnit tests are collected by taking
      1) everything that extends JUnit 3 TestCase
      2) every class that is annotated with JUnit 4 @Test
      3) every method that is annotated with JUnit 4 @Test

      2) is clearly wrong, as @Test only applies to methods, not classes
      http://junit.sourceforge.net/javadoc_40/org/junit/Test.html

      Also, I found no way to run JDave ( http://www.jdave.org/ ) specs or a custom suite with the current 1.3 trunk (22 April 2008). In 1.2, this is very easy with Buildr::Java::JUnit::JUNIT_TESTS_PATTERN filtering by test / spec class names.

      Some kind of an abstraction for filtering the relevant test or spec classes is important not only for custom test framework integration, but for custom suites in big projects. Running a lot of tests is typically faster in a suite, and when you have different sets of tests (unit, integration, acceptance, black-box against a deployed application, ...) you typically want to specify which set to run. For example, the continuous integration server might only run tests against the deployed application a couple of times a day, but unit tests all the time for fast feedback. Different grouping methods of test frameworks are framework specific, and rather than making Buildr support framework specific ways, it could operate on an extra abstraction level.

      Filtering by class name has worked well elsewhere (ant, maven, buildr 1.2), so it would be excellent if it would be included in 1.3 as well.

        Activity

        Hide
        Timo Rantalaiho added a comment -

        By testing @RunWith instead of @Test on class level, JDave integration becomes possible, but this does not yet solve the whole problem (that of custom suites).

        ----------- clip ---------------------------------------------------------------------------------------------------

        Index: lib/buildr/java/test_frameworks.rb
        ===================================================================
        — lib/buildr/java/test_frameworks.rb (revision 649351)
        +++ lib/buildr/java/test_frameworks.rb (working copy)
        @@ -179,7 +179,7 @@
        def tests(dependencies) #:nodoc:
        filter_classes(dependencies,
        :interfaces => %w

        {junit.framework.TestCase}

        ,

        • :class_annotations => %w {org.junit.Test},
          + :class_annotations => %w{org.junit.runner.RunWith},
          :method_annotations => %w{org.junit.Test}

          )
          end

        ----------- /clip ---------------------------------------------------------------------------------------------------

        Show
        Timo Rantalaiho added a comment - By testing @RunWith instead of @Test on class level, JDave integration becomes possible, but this does not yet solve the whole problem (that of custom suites). ----------- clip --------------------------------------------------------------------------------------------------- Index: lib/buildr/java/test_frameworks.rb =================================================================== — lib/buildr/java/test_frameworks.rb (revision 649351) +++ lib/buildr/java/test_frameworks.rb (working copy) @@ -179,7 +179,7 @@ def tests(dependencies) #:nodoc: filter_classes(dependencies, :interfaces => %w {junit.framework.TestCase} , :class_annotations => %w {org.junit.Test}, + :class_annotations => %w{org.junit.runner.RunWith}, :method_annotations => %w{org.junit.Test} ) end ----------- /clip ---------------------------------------------------------------------------------------------------
        Hide
        Assaf Arkin added a comment -

        Right. #2 should be RunWith, not Test.

        Running JDave with the JUnit framework will not be supported in 1.3 or future versions. Rather, we added more abstraction and convenience methods to support easily adding new test frameworks, so JDave should have its own test framework. Since it's BDD, it might be easier to start with TestFramework::JavaBDD (buildr/java/bdd_frameworks.rb).

        It's still possible to select a subset of the available test cases using include/exclude, and these are applied uniformly across all test frameworks.:
        1. The test framework identifies all the tests is can run and provides their name
        2. The include/exclude patterns are applied to select the tests that will actually execute
        3. Integration tests are selected separately from unit tests (integration and test tasks, respectively)
        4. The resulting list is then passed back to the test framework for execution

        Show
        Assaf Arkin added a comment - Right. #2 should be RunWith, not Test. Running JDave with the JUnit framework will not be supported in 1.3 or future versions. Rather, we added more abstraction and convenience methods to support easily adding new test frameworks, so JDave should have its own test framework. Since it's BDD, it might be easier to start with TestFramework::JavaBDD (buildr/java/bdd_frameworks.rb). It's still possible to select a subset of the available test cases using include/exclude, and these are applied uniformly across all test frameworks.: 1. The test framework identifies all the tests is can run and provides their name 2. The include/exclude patterns are applied to select the tests that will actually execute 3. Integration tests are selected separately from unit tests (integration and test tasks, respectively) 4. The resulting list is then passed back to the test framework for execution
        Hide
        Pekka Enberg added a comment - - edited

        One of the biggest strengths of JDave is the fact that it doesn't need any special tools support so I consider dropping this from Buildr 1.3 a serious regression. Especially considering how Buildr doesn't support JDave "natively" I strongly encourage you to merge this patch (or some other version of the patch) as a stop-gap measure so that existing users can upgrade to 1.3.

        Furthermore, keep in mind that @RunWith is not specific to JDave at all so you're potentially breaking other working setups as well here.

        Show
        Pekka Enberg added a comment - - edited One of the biggest strengths of JDave is the fact that it doesn't need any special tools support so I consider dropping this from Buildr 1.3 a serious regression. Especially considering how Buildr doesn't support JDave "natively" I strongly encourage you to merge this patch (or some other version of the patch) as a stop-gap measure so that existing users can upgrade to 1.3. Furthermore, keep in mind that @RunWith is not specific to JDave at all so you're potentially breaking other working setups as well here.
        Hide
        Pekka Enberg added a comment -

        And btw, how do you plan to handle JUnit 4 suites that also use the @RunWith mechanism? You need to be able to include/exclude suites from the Buildr build all the same. So as long as you support JUnit4 properly, there's no reason why JDave wouldn't work.

        Show
        Pekka Enberg added a comment - And btw, how do you plan to handle JUnit 4 suites that also use the @RunWith mechanism? You need to be able to include/exclude suites from the Buildr build all the same. So as long as you support JUnit4 properly, there's no reason why JDave wouldn't work.
        Hide
        Assaf Arkin added a comment -

        What Buildr 1.3 doesn't support is allowing you to use the JUnit test framework to run tests that are not identifiable by either JUnit 3 extension or JUnit 4 annotation. That has caused problems in the past, and we fixed that for 1.3.

        The class annotation being @Test instead of @RunWith is a bug that's getting fixed right now (silly editing error). I'm adding a test case to make sure it will remain @RunWith moving forward.

        Show
        Assaf Arkin added a comment - What Buildr 1.3 doesn't support is allowing you to use the JUnit test framework to run tests that are not identifiable by either JUnit 3 extension or JUnit 4 annotation. That has caused problems in the past, and we fixed that for 1.3. The class annotation being @Test instead of @RunWith is a bug that's getting fixed right now (silly editing error). I'm adding a test case to make sure it will remain @RunWith moving forward.
        Hide
        Pekka Enberg added a comment -

        Okay, as long as we can include/exclude specific suites annotated with @RunWith, existing JDave and JUnit4 will work just fine. Thanks.

        Show
        Pekka Enberg added a comment - Okay, as long as we can include/exclude specific suites annotated with @RunWith, existing JDave and JUnit4 will work just fine. Thanks.
        Hide
        Assaf Arkin added a comment -

        I just committed this fix to SVN and also verified that the following will work:

        1. JUnit will pick any class that has one or more @Test annotated method in it.
        2. JUnit will also pick any class that has @RunWith annotation on it, or extends TestCase (for JUnit 3.x support).
        3. When using test.include('foo') only the name 'foo' is passed to JUnit for execution, provided that 'foo' is either #1 or #2.

        Show
        Assaf Arkin added a comment - I just committed this fix to SVN and also verified that the following will work: 1. JUnit will pick any class that has one or more @Test annotated method in it. 2. JUnit will also pick any class that has @RunWith annotation on it, or extends TestCase (for JUnit 3.x support). 3. When using test.include('foo') only the name 'foo' is passed to JUnit for execution, provided that 'foo' is either #1 or #2.
        Hide
        Timo Rantalaiho added a comment -

        Thanks Assaf, with @RunWith + include / exclude things seem to work!

        Show
        Timo Rantalaiho added a comment - Thanks Assaf, with @RunWith + include / exclude things seem to work!

          People

          • Assignee:
            Unassigned
            Reporter:
            Timo Rantalaiho
          • Votes:
            10 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development