Maven Surefire
  1. Maven Surefire
  2. SUREFIRE-745

-Dtest supports multiple test classes but not multiple test methods

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.12.1, 2.19
    • Component/s: None
    • Labels:
      None
    • Environment:

      Description

      The -Dtest parameter is very handy for running a specific test class or test method. It also supports running multiple test classes. Unfortunately, it does not permit specifying running multiple test methods. It would be great if this were possible.

      The examples below are from the Apache Commons project.

      WORKS: Run multiple test classes:
      mvn test -Dtest=ImmutablePairTest,StopWatchTest

      WORKS: Run a specific test method:
      mvn test -Dtest=ImmutablePairTest#testBasic

      DOES NOT WORK:
      mvn test -Dtest=StopWatchTest#testStopWatchSimple,StopWatchTest#testStopWatchSimpleGet
      mvn test -Dtest=ImmutablePairTest#testBasic,StopWatchTest#testLang315

      1. multipleMethods.patch
        10 kB
        rainLee
      2. multipleMethods-v2.patch
        5 kB
        rainLee
      3. multipleMethods-v3.patch
        14 kB
        rainLee
      4. multipleMethods-v4.patch
        14 kB
        rainLee
      5. SUREFIRE-745.patch
        18 kB
        rainLee
      6. SUREFIRE-745-v2.patch
        22 kB
        rainLee

        Activity

        Reid Holmes created issue -
        Olivier Lamy (*$^¨%`£) made changes -
        Field Original Value New Value
        Assignee Olivier Lamy [ olamy ]
        Hide
        rainLee added a comment -

        i have fixed this issue
        can i submit a patch?

        Show
        rainLee added a comment - i have fixed this issue can i submit a patch?
        Hide
        Olivier Lamy (*$^¨%`£) added a comment -

        sure
        with it test integrated ?

        Show
        Olivier Lamy (*$^¨%`£) added a comment - sure with it test integrated ?
        Hide
        rainLee added a comment -

        my patch is created from version 2.8
        should i created from trunk?

        Show
        rainLee added a comment - my patch is created from version 2.8 should i created from trunk?
        Hide
        rainLee added a comment - - edited

        submit my patch that created from version 2.8
        and my patch support -Dtest=xxx.xxx.ATest#testMethod1#testMethod2,xxx.xxx.BTest#testMethod3

        Show
        rainLee added a comment - - edited submit my patch that created from version 2.8 and my patch support -Dtest=xxx.xxx.ATest#testMethod1#testMethod2,xxx.xxx.BTest#testMethod3
        rainLee made changes -
        Attachment multipleMethods.patch [ 58961 ]
        Hide
        Kristian Rosenvold added a comment -

        Patches should always be created from trunk.

        Show
        Kristian Rosenvold added a comment - Patches should always be created from trunk.
        Hide
        rainLee added a comment -

        update my patch.
        this patch is created from trunk

        Show
        rainLee added a comment - update my patch. this patch is created from trunk
        rainLee made changes -
        Attachment multipleMethods-v2.patch [ 58964 ]
        Hide
        rainLee added a comment -

        add integrated test

        Show
        rainLee added a comment - add integrated test
        rainLee made changes -
        Attachment multipleMethods-v3.patch [ 58965 ]
        Hide
        Robert Scholte added a comment -

        I noticed that mtClass#myMethod#myMethod2,secondClass#testMethod would become a valid expression.
        IMO the # is a class-to-method separator. My suggestion would be to use + between methods:
        mtClass#myMethod+myMethod2,secondClass#testMethod

        Show
        Robert Scholte added a comment - I noticed that mtClass#myMethod#myMethod2,secondClass#testMethod would become a valid expression. IMO the # is a class-to-method separator. My suggestion would be to use + between methods: mtClass#myMethod+myMethod2,secondClass#testMethod
        Hide
        rainLee added a comment -

        should I take Robert's opinion?

        Show
        rainLee added a comment - should I take Robert's opinion?
        Hide
        Olivier Lamy (*$^¨%`£) added a comment -

        agree with Robert.
        BTW mtClass#myMethod* must still work and execute both myMethod AND myMethod2.
        I think there is an integration test for that.

        Show
        Olivier Lamy (*$^¨%`£) added a comment - agree with Robert. BTW mtClass#myMethod* must still work and execute both myMethod AND myMethod2. I think there is an integration test for that.
        Hide
        rainLee added a comment -

        yeah, Olivier.
        mtClass#myMethod* does still work and the integration test is passed

        ok, i'll take Robert's opinion and refresh the patch later

        Show
        rainLee added a comment - yeah, Olivier. mtClass#myMethod* does still work and the integration test is passed ok, i'll take Robert's opinion and refresh the patch later
        Hide
        rainLee added a comment -

        change method-method separator from "#" to "+"

        Show
        rainLee added a comment - change method-method separator from "#" to "+"
        rainLee made changes -
        Attachment multipleMethods-v4.patch [ 58996 ]
        Hide
        Olivier Lamy (*$^¨%`£) added a comment -

        can you generate a patch from top level using svn diff > SUREFIRE-745.patch
        It looks you generate this with eclipse which means I cannot apply it using cli (as I use intellij )
        And sorry to bother you .

        Show
        Olivier Lamy (*$^¨%`£) added a comment - can you generate a patch from top level using svn diff > SUREFIRE-745 .patch It looks you generate this with eclipse which means I cannot apply it using cli (as I use intellij ) And sorry to bother you .
        Hide
        rainLee added a comment -

        sure.
        sorry to make you wield.cause this is my first time submit a patch to the community.
        I should have known this before i submit.
        thanks for reminding me

        Show
        rainLee added a comment - sure. sorry to make you wield.cause this is my first time submit a patch to the community. I should have known this before i submit. thanks for reminding me
        rainLee made changes -
        Attachment SUREFIRE-745.patch [ 58997 ]
        Hide
        Olivier Lamy (*$^¨%`£) added a comment -

        why all of those changes:

           <parent>
             <groupId>org.apache.maven.surefire</groupId>
             <artifactId>surefire-providers</artifactId>
        -    <version>2.8</version>
        +    <version>2.8-itest</version>
             <relativePath>../pom.xml</relativePath>
           </parent>
        
        Show
        Olivier Lamy (*$^¨%`£) added a comment - why all of those changes: <parent> <groupId>org.apache.maven.surefire</groupId> <artifactId>surefire-providers</artifactId> - <version>2.8</version> + <version>2.8-itest</version> <relativePath>../pom.xml</relativePath> </parent>
        Hide
        rainLee added a comment -

        oh,sorry,i have already used this feature in my own maven repository,so i changed the version.
        i'll revert this soon

        Show
        rainLee added a comment - oh,sorry,i have already used this feature in my own maven repository,so i changed the version. i'll revert this soon
        Hide
        Olivier Lamy (*$^¨%`£) added a comment -

        As I see only it test for junit 4.4: does that work with all junit 4.x versions ?

        Show
        Olivier Lamy (*$^¨%`£) added a comment - As I see only it test for junit 4.4: does that work with all junit 4.x versions ?
        Hide
        rainLee added a comment -

        update patch
        remove pom modification
        add junit48 integration test

        Show
        rainLee added a comment - update patch remove pom modification add junit48 integration test
        rainLee made changes -
        Attachment SUREFIRE-745-v2.patch [ 59007 ]
        Hide
        Olivier Lamy (*$^¨%`£) added a comment - - edited

        fixed r1294568.
        Thanks !

        Show
        Olivier Lamy (*$^¨%`£) added a comment - - edited fixed r1294568. Thanks !
        Olivier Lamy (*$^¨%`£) made changes -
        Resolution Fixed [ 1 ]
        Status Open [ 1 ] Closed [ 6 ]
        Fix Version/s 2.13 [ 18312 ]
        Hide
        rainLee added a comment -

        my pleasure

        Show
        rainLee added a comment - my pleasure
        Hide
        Jianfeng Sun added a comment -

        I did try multiple methods from different classes on Maven 3.x, Surefire Failsafe 2.17 with TestNG 6.8.x. But it was still not working by now.
        According to SureFire website http://maven.apache.org/surefire/maven-failsafe-plugin/examples/single-test.html, it looks like multiple method will only be supported with JUnit 4.x. Will some one add a patch for TestNG?

        Show
        Jianfeng Sun added a comment - I did try multiple methods from different classes on Maven 3.x, Surefire Failsafe 2.17 with TestNG 6.8.x. But it was still not working by now. According to SureFire website http://maven.apache.org/surefire/maven-failsafe-plugin/examples/single-test.html , it looks like multiple method will only be supported with JUnit 4.x. Will some one add a patch for TestNG?
        Hide
        Tibor Digana added a comment -

        @Jianfeng Sun
        Can you provide a patch in GitHub with TestNG method selector filtering methods per class?

        Show
        Tibor Digana added a comment - @Jianfeng Sun Can you provide a patch in GitHub with TestNG method selector filtering methods per class?
        Hide
        Jianfeng Sun added a comment - - edited

        @ Tibor Digana
        Today, I tried this with JUnit and I saw same issue. I manually specifying a provider
        <dependency>
        <groupId>org.apache.maven.surefire</groupId>
        <artifactId>surefire-junit47</artifactId>
        <version>2.17</version>
        </dependency>

        1) mvn -Dit.test=class1#method1,class2#method2 verify
        get 0 test case error
        2) mvn -Dit.test=class1,class2#method2 verify
        get a) only method 2 executed
        b) class1 + method1 from class2 and method2 from class2 3 test executed
        3) mvn -Dit.test=class1#method1,class2#* verify
        get 0 test case error

        Is this an issue or I made any mistake here?

        +and I need to mention that I'm using failsafe 2.17 not surefire even if I don't think that will be an issue.

        Show
        Jianfeng Sun added a comment - - edited @ Tibor Digana Today, I tried this with JUnit and I saw same issue. I manually specifying a provider <dependency> <groupId>org.apache.maven.surefire</groupId> <artifactId>surefire-junit47</artifactId> <version>2.17</version> </dependency> 1) mvn -Dit.test=class1#method1,class2#method2 verify get 0 test case error 2) mvn -Dit.test=class1,class2#method2 verify get a) only method 2 executed b) class1 + method1 from class2 and method2 from class2 3 test executed 3) mvn -Dit.test=class1#method1,class2#* verify get 0 test case error Is this an issue or I made any mistake here? +and I need to mention that I'm using failsafe 2.17 not surefire even if I don't think that will be an issue.
        Hide
        Tibor Digana added a comment -

        @Jianfeng Sun
        Have you tried with system property -Dtest=... and the same provider ?

        Show
        Tibor Digana added a comment - @Jianfeng Sun Have you tried with system property -Dtest=... and the same provider ?
        Hide
        Tibor Digana added a comment -

        @Jianfeng Sun
        We have fixed such bug https://jira.codehaus.org/browse/SUREFIRE-1028 in 2.18-SNAPSHOT.
        Pls setup repository http://repository.apache.org/content/groups/snapshots for the SNAPSHOT version of maven-surefire.
        Let us know if it works for you.

        Show
        Tibor Digana added a comment - @Jianfeng Sun We have fixed such bug https://jira.codehaus.org/browse/SUREFIRE-1028 in 2.18-SNAPSHOT. Pls setup repository http://repository.apache.org/content/groups/snapshots for the SNAPSHOT version of maven-surefire. Let us know if it works for you.
        Hide
        Jianfeng Sun added a comment - - edited

        @Tibor

        Thanks for the response.

        By using 2.18-SNAPSHOT for both surefire and failsafe,
        1) if I use -Dit.test=class1#method1,class2#method2
        I got this:
        main:
        [INFO] Executed tasks
        [INFO]
        [INFO] — maven-surefire-plugin:2.18-SNAPSHOT:test (default-test) @ selenium —
        [INFO]
        [INFO] — maven-jar-plugin:2.4:jar (default-jar) @ selenium —
        [INFO]
        [INFO] — maven-failsafe-plugin:2.18-SNAPSHOT:integration-test (run-test) @ selenium —
        [INFO] Failsafe report directory: C:\Automation testing tool\workspace\fox.emf.selenium.Oct5\target\surefire-reports\10-15-2014-03-55-PM
        [INFO] Using configured provider org.apache.maven.surefire.junitcore.JUnitCoreProvider
        [INFO] parallel='none', perCoreThreadCount=true, threadCount=0, useUnlimitedThreads=false, threadCountSuites=0, threadCountClasses=0, threadCountMethods=0, parallelOptimized=true

        -------------------------------------------------------
        T E S T S
        -------------------------------------------------------

        -------------------------------------------------------
        T E S T S
        -------------------------------------------------------

        Results :

        Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

        [WARNING] File encoding has not been set, using platform encoding Cp1252, i.e. build is platform dependent!
        [INFO]
        [INFO] — properties-maven-plugin:1.0-alpha-2:read-project-properties (default) @ selenium —
        [INFO]
        [INFO] — maven-postman-plugin:0.1.6:send-mail (send_email) @ selenium —

        2) if I use -Dtest=class1#method1,class2#method2
        I got Failed tests:
        class1.method1:21 ? NullPointer
        class1.method2:16 ? NullPointer
        class2.method1:21 ? NullPointer
        class2.method2:17 ? NullPointer

        the java code for NullPointer is "System.getProperty("user.dir")+"/testflow/Spot_TC07_Export Functionality.xls""
        I guess the NullPorinter is from system property user dir.
        I'm sure it will work when we only run one test at a time.

        Show
        Jianfeng Sun added a comment - - edited @Tibor Thanks for the response. By using 2.18-SNAPSHOT for both surefire and failsafe, 1) if I use -Dit.test=class1#method1,class2#method2 I got this: main: [INFO] Executed tasks [INFO] [INFO] — maven-surefire-plugin:2.18-SNAPSHOT:test (default-test) @ selenium — [INFO] [INFO] — maven-jar-plugin:2.4:jar (default-jar) @ selenium — [INFO] [INFO] — maven-failsafe-plugin:2.18-SNAPSHOT:integration-test (run-test) @ selenium — [INFO] Failsafe report directory: C:\Automation testing tool\workspace\fox.emf.selenium.Oct5\target\surefire-reports\10-15-2014-03-55-PM [INFO] Using configured provider org.apache.maven.surefire.junitcore.JUnitCoreProvider [INFO] parallel='none', perCoreThreadCount=true, threadCount=0, useUnlimitedThreads=false, threadCountSuites=0, threadCountClasses=0, threadCountMethods=0, parallelOptimized=true ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Results : Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 [WARNING] File encoding has not been set, using platform encoding Cp1252, i.e. build is platform dependent! [INFO] [INFO] — properties-maven-plugin:1.0-alpha-2:read-project-properties (default) @ selenium — [INFO] [INFO] — maven-postman-plugin:0.1.6:send-mail (send_email) @ selenium — 2) if I use -Dtest=class1#method1,class2#method2 I got Failed tests: class1.method1:21 ? NullPointer class1.method2:16 ? NullPointer class2.method1:21 ? NullPointer class2.method2:17 ? NullPointer the java code for NullPointer is "System.getProperty("user.dir")+"/testflow/Spot_TC07_Export Functionality.xls"" I guess the NullPorinter is from system property user dir. I'm sure it will work when we only run one test at a time.
        Hide
        Tibor Digana added a comment -

        @Jianfeng Sun
        Regarding filters (classes & methods) and groups my plan is to build IT tests with surefire-junit47 in 2.18.
        If they fail I will report and fix a bug so we will have chance to get stable surefire plugin.
        Did you combine "parallel=..." with surefire in the above tests ?

        Show
        Tibor Digana added a comment - @Jianfeng Sun Regarding filters (classes & methods) and groups my plan is to build IT tests with surefire-junit47 in 2.18. If they fail I will report and fix a bug so we will have chance to get stable surefire plugin. Did you combine "parallel=..." with surefire in the above tests ?
        Hide
        Jianfeng Sun added a comment - - edited

        @Tibo Digana
        Filters and groups will be great features for surefire. Thank you for the work.
        I didn't use parallel parameter in this test. So I believe by default it should be set as "false".

        And I just tried

        <configuration>
        <parallel>methods</parallel>
        <threadCount>10</threadCount>
        </configuration>

        see the same result.

        Show
        Jianfeng Sun added a comment - - edited @Tibo Digana Filters and groups will be great features for surefire. Thank you for the work. I didn't use parallel parameter in this test. So I believe by default it should be set as "false". And I just tried <configuration> <parallel>methods</parallel> <threadCount>10</threadCount> </configuration> see the same result.
        Hide
        Tibor Digana added a comment -

        Multiple Classes and Methods, regex, non-regex support for JUnit4, JUnit47+, TestNG providers.
        https://github.com/apache/maven-surefire/pull/83

        Show
        Tibor Digana added a comment - Multiple Classes and Methods, regex, non-regex support for JUnit4, JUnit47+, TestNG providers. https://github.com/apache/maven-surefire/pull/83
        Tibor Digana made changes -
        Resolution Fixed [ 1 ]
        Status Closed [ 6 ] Reopened [ 4 ]
        Assignee Olivier Lamy [ olamy ] Tibor Digana [ tibor17 ]
        Tibor Digana made changes -
        Fix Version/s 2.19 [ 20728 ]
        Hide
        Tibor Digana added a comment -

        commit 866a535b66229e9bab3f09cbd42732d70ca9504f
        commit 947abde38a101b028deabf0316d5efa44dd074b7
        commit ca9eb3d1a344813457a959481c3ff73bde074721
        commit 2824f9408e751b383770a77d260a07d9f1d7e544

        Show
        Tibor Digana added a comment - commit 866a535b66229e9bab3f09cbd42732d70ca9504f commit 947abde38a101b028deabf0316d5efa44dd074b7 commit ca9eb3d1a344813457a959481c3ff73bde074721 commit 2824f9408e751b383770a77d260a07d9f1d7e544
        Tibor Digana made changes -
        Resolution Fixed [ 1 ]
        Status Reopened [ 4 ] Closed [ 6 ]
        Mark Thomas made changes -
        Project Import Sun Apr 05 13:23:32 UTC 2015 [ 1428240212200 ]
        Mark Thomas made changes -
        Workflow jira [ 12727991 ] Default workflow, editable Closed status [ 12759083 ]
        Mark Thomas made changes -
        Project Import Mon Apr 06 01:36:33 UTC 2015 [ 1428284193036 ]
        Mark Thomas made changes -
        Workflow jira [ 12965890 ] Default workflow, editable Closed status [ 13003546 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Closed Closed
        258d 15h 52m 1 Olivier Lamy (*$^¨%`£) 28/Feb/12 04:47
        Closed Closed Reopened Reopened
        1120d 13h 49m 1 Tibor Digana 24/Mar/15 18:37
        Reopened Reopened Closed Closed
        1d 1h 7m 1 Tibor Digana 25/Mar/15 19:44

          People

          • Assignee:
            Tibor Digana
            Reporter:
            Reid Holmes
          • Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development