Maven Surefire
  1. Maven Surefire
  2. SUREFIRE-530

Allow runtime ordering of tests to be specified

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.4.3
    • Fix Version/s: 3.0
    • Component/s: Maven Surefire Plugin
    • Labels:
      None
    • Environment:
      testNG, Junit

      Description

      Allow other plugins to specify tests to get run, and preserve the ordering of tests.

      Currently, a plugin may set the "test" parameter with a comma separated list of tests to run.
      However, the order is not preserved.

      For plugins such as the maven-clover2-plugin, (possibly the maven-reactor-plugin?) that optimize the build/test run, ordering of a tests is a very nice way to ensure the build fails as fast as possible.

        Issue Links

          Activity

          Nick Pellow created issue -
          Nick Pellow made changes -
          Field Original Value New Value
          Link This issue is related to SUREFIRE-526 [ SUREFIRE-526 ]
          Kristian Rosenvold made changes -
          Link This issue relates to SUREFIRE-726 [ SUREFIRE-726 ]
          Hide
          Tibor Digana added a comment -

          @Nick
          Did you try to use the runOrder? It would not help?

          Show
          Tibor Digana added a comment - @Nick Did you try to use the runOrder? It would not help?
          Mark Thomas made changes -
          Project Import Sun Apr 05 13:23:32 UTC 2015 [ 1428240212200 ]
          Mark Thomas made changes -
          Workflow jira [ 12727781 ] Default workflow, editable Closed status [ 12758411 ]
          Mark Thomas made changes -
          Project Import Mon Apr 06 01:36:33 UTC 2015 [ 1428284193036 ]
          Mark Thomas made changes -
          Workflow jira [ 12965352 ] Default workflow, editable Closed status [ 13002558 ]
          Hide
          Christian added a comment - - edited

          I am currently looking for a way to specify the order of the tests as well. Has there been any update?

          @Tibor Digana The parameter runOrder does not seem to be an option, since it does not allow a custom ordering of test classes, but only some predefined orderings.

          Show
          Christian added a comment - - edited I am currently looking for a way to specify the order of the tests as well. Has there been any update? @ Tibor Digana The parameter runOrder does not seem to be an option, since it does not allow a custom ordering of test classes, but only some predefined orderings.
          Hide
          Tibor Digana added a comment -

          Christian
          Regarding surefire fix we want to write extension (programming) API to customize surefire behavior. Can you make any suggestion about the API? This is the branch with some starting point of the extensions based on SPI
          https://github.com/apache/maven-surefire/tree/s1

          Can you specify a format of test order including the API?
          Since the textual order may not satisfy requirements of all communities, I prefer soft coded API and later we can allow users type Groovy scripts.

          Show
          Tibor Digana added a comment - Christian Regarding surefire fix we want to write extension (programming) API to customize surefire behavior. Can you make any suggestion about the API? This is the branch with some starting point of the extensions based on SPI https://github.com/apache/maven-surefire/tree/s1 Can you specify a format of test order including the API? Since the textual order may not satisfy requirements of all communities, I prefer soft coded API and later we can allow users type Groovy scripts.
          Hide
          Christian added a comment - - edited

          @Tibor Digana Thanks for your quick reply and the great work on SureFire so far

          My original idea was just to have a parameter similar to excludes and includes:

          <order>
            <orderItem>ExampleATest</orderItem>
            <orderItem>ExampleCTest</orderItem>
            <orderItem>ExampleBTest</orderItem>
            <orderItem>ExampleDTest</orderItem>
          </order>
          

          In terms of an API, I assume you want the API user to be able to configure and trigger surefire as well as getting the results of the test run, right? In this case, I think it would be nice, to be able, to somehow pass a list of tests, that are run with respect to the given order. Since I have only a vague idea of what you want to achieve, I find it difficult to be more specific.

          Regarding the s1 branch: I spend the last two days digging into SureFire's architecture, but I do not really have an overview yet. Thus, I cannot easily see, what you are doing in the s1 branch. Can you maybe point out the API relevant files and how you intend your API to be called?

          Show
          Christian added a comment - - edited @ Tibor Digana Thanks for your quick reply and the great work on SureFire so far My original idea was just to have a parameter similar to excludes and includes : <order> <orderItem>ExampleATest</orderItem> <orderItem>ExampleCTest</orderItem> <orderItem>ExampleBTest</orderItem> <orderItem>ExampleDTest</orderItem> </order> In terms of an API, I assume you want the API user to be able to configure and trigger surefire as well as getting the results of the test run, right? In this case, I think it would be nice, to be able, to somehow pass a list of tests, that are run with respect to the given order. Since I have only a vague idea of what you want to achieve, I find it difficult to be more specific. Regarding the s1 branch: I spend the last two days digging into SureFire's architecture, but I do not really have an overview yet. Thus, I cannot easily see, what you are doing in the s1 branch. Can you maybe point out the API relevant files and how you intend your API to be called?
          Hide
          Tibor Digana added a comment -

          Assigned to 3.0.
          We will write the API documentation and my wish is to use script language to come over building customer's artifact and simplify the use.
          The problem is that plugin configuration is getting too big.
          Run-order functionality needs some work to be present in configuration. Christian you can commit PR with a patch on GitHub and we will use it in 3.0 which will speed up our work.

          Show
          Tibor Digana added a comment - Assigned to 3.0. We will write the API documentation and my wish is to use script language to come over building customer's artifact and simplify the use. The problem is that plugin configuration is getting too big. Run-order functionality needs some work to be present in configuration. Christian you can commit PR with a patch on GitHub and we will use it in 3.0 which will speed up our work.
          Tibor Digana made changes -
          Fix Version/s 3.0 [ 12331667 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Nick Pellow
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development