Maven Surefire
  1. Maven Surefire
  2. SUREFIRE-580

Allow "fail fast" or stop running on first failure

    Details

      Description

      We have some slow Selenium builds. Waiting for indication of failure can take a long time. It would be helpful if there was an option to finish the test run on the first failure.

      We could configure this on some of our slower builds so improve the speed of the feedback loop in the cases where there are failures.

        Activity

        Paul Curren created issue -
        Hide
        Deejay added a comment - - edited

        I also require this feature. When using a combination of Spring and Liquibase, if the Spring config contains an error then on each and every test the Spring context will try once more to initialize itself, taking 20+ seconds each time.

        This adds up to a huge amount of time wasted by our Jenkins instance to alert us to the fact that someone made a typo.

        Show
        Deejay added a comment - - edited I also require this feature. When using a combination of Spring and Liquibase, if the Spring config contains an error then on each and every test the Spring context will try once more to initialize itself, taking 20+ seconds each time. This adds up to a huge amount of time wasted by our Jenkins instance to alert us to the fact that someone made a typo.
        Hide
        Tomislav Nakic-Alfirevic added a comment -

        Same problem here: we use surefire + junit to run integration tests (maybe there is a better way, I don't know...) and our test suite takes hours to complete right now, yet most of the time we expect 100% correct execution and would like to know - as soon as possible - which test failed and why.

        I half expect I'm using the wrong tool or using the tool wrong and one or more simple solutions are available. That said, I'm a bit puzzled that this issue remains open for three years: it seems to me enough to add a switch (e.g. -DfastFail=true, default=false) and wrap up test execution as soon as the first test fails, producing the usual test run output. Test methods which haven't been executed might have to be removed from internal data structures so that they do not appear in the reports.

        Show
        Tomislav Nakic-Alfirevic added a comment - Same problem here: we use surefire + junit to run integration tests (maybe there is a better way, I don't know...) and our test suite takes hours to complete right now, yet most of the time we expect 100% correct execution and would like to know - as soon as possible - which test failed and why. I half expect I'm using the wrong tool or using the tool wrong and one or more simple solutions are available. That said, I'm a bit puzzled that this issue remains open for three years: it seems to me enough to add a switch (e.g. -DfastFail=true, default=false) and wrap up test execution as soon as the first test fails, producing the usual test run output. Test methods which haven't been executed might have to be removed from internal data structures so that they do not appear in the reports.
        Hide
        Kristian Rosenvold added a comment -

        Could probably be hooked up to the maven "-ff" option if someone is willing to make the patch

        Show
        Kristian Rosenvold added a comment - Could probably be hooked up to the maven "-ff" option if someone is willing to make the patch
        Hide
        Tomislav Nakic-Alfirevic added a comment -

        It would even be useful if test execution was not interrupted, but the result of each test be output after the test, not after the entire test suite. Something that would work like this:

        $ mvn test | grep FAIL
        SuiteA.test2 FAILED
        SuiteA.test11 FAILED
        SuiteB.test1 FAILED
        [surefire still running]

        I imagine that should be fairly easy to implement, yet it would still provide quick information about any test failures...

        Show
        Tomislav Nakic-Alfirevic added a comment - It would even be useful if test execution was not interrupted, but the result of each test be output after the test, not after the entire test suite. Something that would work like this: $ mvn test | grep FAIL SuiteA.test2 FAILED SuiteA.test11 FAILED SuiteB.test1 FAILED [surefire still running] I imagine that should be fairly easy to implement, yet it would still provide quick information about any test failures...
        Hide
        Gili added a comment -

        Please add TestNG to the list of components. This should be fixed for both frameworks.

        Show
        Gili added a comment - Please add TestNG to the list of components. This should be fixed for both frameworks.
        Andreas Gudian made changes -
        Field Original Value New Value
        Component/s TestNG support [ 12346 ]
        Hide
        J Knurek added a comment -

        I've submitted a pull-request in case anyone wants to work with it and get this implemented:
        https://github.com/apache/maven-surefire/pull/37

        Show
        J Knurek added a comment - I've submitted a pull-request in case anyone wants to work with it and get this implemented: https://github.com/apache/maven-surefire/pull/37
        Tibor Digana made changes -
        Fix Version/s 2.19 [ 20728 ]
        Hide
        Tibor Digana added a comment -

        @Kristian
        I will try to implement it with your proposal
        MavenExecutionRequest.REACTOR_FAIL_FAST.equals( mavenSession.getReactorFailureBehavior() )

        Show
        Tibor Digana added a comment - @Kristian I will try to implement it with your proposal MavenExecutionRequest.REACTOR_FAIL_FAST.equals( mavenSession.getReactorFailureBehavior() )
        Mark Thomas made changes -
        Project Import Sun Apr 05 13:23:32 UTC 2015 [ 1428240212200 ]
        Mark Thomas made changes -
        Workflow jira [ 12727829 ] Default workflow, editable Closed status [ 12758992 ]
        Mark Thomas made changes -
        Project Import Mon Apr 06 01:36:33 UTC 2015 [ 1428284193036 ]
        Mark Thomas made changes -
        Workflow jira [ 12965830 ] Default workflow, editable Closed status [ 13003477 ]
        Tibor Digana made changes -
        Assignee Tibor Digana [ tibor17 ]
        Hide
        Tibor Digana added a comment -

        I first implemented retrieval of CLI Options of the main process and I will fix SUREFIRE-1158 and this issue which need common functionality.

        Show
        Tibor Digana added a comment - I first implemented retrieval of CLI Options of the main process and I will fix SUREFIRE-1158 and this issue which need common functionality.
        Tibor Digana made changes -
        Component/s Junit 4.7+ (parallel) support [ 12326510 ]
        Show
        Tibor Digana added a comment - https://github.com/apache/maven-surefire/commit/78dca27ef03bd939e700d7b00babaac44e7d1707 SUREFIRE-580 SUREFIRE-524 new mechanism to dispatch commands to forked jvm
        Hide
        ASF GitHub Bot added a comment -

        GitHub user Tibor17 opened a pull request:

        https://github.com/apache/maven-surefire/pull/103

        SUREFIRE-580 Allow "fail fast" or stop running on first failure

        Solution for https://issues.apache.org/jira/browse/SUREFIRE-580
        Refactoring
        and InputStream in forked jvm is command based which helps us to fix SUREFIRE-524.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/Tibor17/maven-surefire s2

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/maven-surefire/pull/103.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #103


        commit 4aa1424443066f165f17cc0fd3b3b701719c8b7f
        Author: Tibor17 <tibor17@lycos.com>
        Date: 2015-07-24T21:22:35Z

        [SUREFIRE] added missing interface MainCliOptionsAware

        commit b3f53ceeacc2c49a1256e660da0b3fdd5ac0eefa
        Author: Tibor17 <tibor17@lycos.com>
        Date: 2015-07-25T01:28:20Z

        [SUREFIRE] added missing interface MainCliOptionsAware

        commit 84b2be1144e9fe0366deeab1ed341d6b0937ae31
        Author: Tibor17 <tibor17@lycos.com>
        Date: 2015-07-25T01:30:55Z

        [SUREFIRE] MasterProcessReader Thread should be quite after throwing exception

        commit 7ba216073200bfd295d4e2388e25a9958f9e3f35
        Author: Tibor17 <tibor17@lycos.com>
        Date: 2015-08-03T13:40:59Z

        [SUREFIRE] refactoring

        commit 7dc69fcd66d38b27dbe44a8e8112dd532e63a990
        Author: Tibor17 <tibor17@lycos.com>
        Date: 2015-08-03T20:48:36Z

        [SUREFIRE] refactoring

        commit 1902155a09325c6df91f42fbf4bba4da66fa5fa4
        Author: Tibor17 <tibor17@lycos.com>
        Date: 2015-08-04T22:00:28Z

        [SUREFIRE] refactoring

        commit 6d15520301ea7a928187ff9e0eb8109660265d29
        Author: Tibor17 <tibor17@lycos.com>
        Date: 2015-09-02T23:33:26Z

        SUREFIRE-580 Allow "fail fast" or stop running on first failure


        Show
        ASF GitHub Bot added a comment - GitHub user Tibor17 opened a pull request: https://github.com/apache/maven-surefire/pull/103 SUREFIRE-580 Allow "fail fast" or stop running on first failure Solution for https://issues.apache.org/jira/browse/SUREFIRE-580 Refactoring and InputStream in forked jvm is command based which helps us to fix SUREFIRE-524 . You can merge this pull request into a Git repository by running: $ git pull https://github.com/Tibor17/maven-surefire s2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/maven-surefire/pull/103.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #103 commit 4aa1424443066f165f17cc0fd3b3b701719c8b7f Author: Tibor17 <tibor17@lycos.com> Date: 2015-07-24T21:22:35Z [SUREFIRE] added missing interface MainCliOptionsAware commit b3f53ceeacc2c49a1256e660da0b3fdd5ac0eefa Author: Tibor17 <tibor17@lycos.com> Date: 2015-07-25T01:28:20Z [SUREFIRE] added missing interface MainCliOptionsAware commit 84b2be1144e9fe0366deeab1ed341d6b0937ae31 Author: Tibor17 <tibor17@lycos.com> Date: 2015-07-25T01:30:55Z [SUREFIRE] MasterProcessReader Thread should be quite after throwing exception commit 7ba216073200bfd295d4e2388e25a9958f9e3f35 Author: Tibor17 <tibor17@lycos.com> Date: 2015-08-03T13:40:59Z [SUREFIRE] refactoring commit 7dc69fcd66d38b27dbe44a8e8112dd532e63a990 Author: Tibor17 <tibor17@lycos.com> Date: 2015-08-03T20:48:36Z [SUREFIRE] refactoring commit 1902155a09325c6df91f42fbf4bba4da66fa5fa4 Author: Tibor17 <tibor17@lycos.com> Date: 2015-08-04T22:00:28Z [SUREFIRE] refactoring commit 6d15520301ea7a928187ff9e0eb8109660265d29 Author: Tibor17 <tibor17@lycos.com> Date: 2015-09-02T23:33:26Z SUREFIRE-580 Allow "fail fast" or stop running on first failure
        Hide
        ASF GitHub Bot added a comment -

        Github user Tibor17 commented on the pull request:

        https://github.com/apache/maven-surefire/pull/103#issuecomment-138125021

        This PR has been merged with master.

        Show
        ASF GitHub Bot added a comment - Github user Tibor17 commented on the pull request: https://github.com/apache/maven-surefire/pull/103#issuecomment-138125021 This PR has been merged with master.
        Tibor Digana made changes -
        Status Open [ 1 ] Closed [ 6 ]
        Resolution Fixed [ 1 ]
        Hide
        Tibor Digana added a comment -

        This feature has been implemented with new configuration parameter "skipAfterFailureCount".

        Show
        Tibor Digana added a comment - This feature has been implemented with new configuration parameter "skipAfterFailureCount".
        Hide
        ASF GitHub Bot added a comment -

        Github user Tibor17 closed the pull request at:

        https://github.com/apache/maven-surefire/pull/103

        Show
        ASF GitHub Bot added a comment - Github user Tibor17 closed the pull request at: https://github.com/apache/maven-surefire/pull/103
        Hide
        ASF GitHub Bot added a comment -

        Github user winterbe commented on the pull request:

        https://github.com/apache/maven-surefire/pull/103#issuecomment-145492582

        Will `skipAfterFailureCount ` also be available in failsafe-plugin or only for surefire-plugin?

        Show
        ASF GitHub Bot added a comment - Github user winterbe commented on the pull request: https://github.com/apache/maven-surefire/pull/103#issuecomment-145492582 Will `skipAfterFailureCount ` also be available in failsafe-plugin or only for surefire-plugin?
        Hide
        ASF GitHub Bot added a comment -

        Github user Tibor17 commented on the pull request:

        https://github.com/apache/maven-surefire/pull/103#issuecomment-145497939

        It will.

        On Mon, Oct 5, 2015 at 12:43 PM, Benjamin Winterberg <
        notifications@github.com> wrote:

        > Will skipAfterFailureCount also be available in failsafe-plugin or only
        > for surefire-plugin?
        >
        > —
        > Reply to this email directly or view it on GitHub
        > <https://github.com/apache/maven-surefire/pull/103#issuecomment-145492582>
        > .
        >


        Cheers
        Tibor

        Show
        ASF GitHub Bot added a comment - Github user Tibor17 commented on the pull request: https://github.com/apache/maven-surefire/pull/103#issuecomment-145497939 It will. On Mon, Oct 5, 2015 at 12:43 PM, Benjamin Winterberg < notifications@github.com> wrote: > Will skipAfterFailureCount also be available in failsafe-plugin or only > for surefire-plugin? > > — > Reply to this email directly or view it on GitHub > < https://github.com/apache/maven-surefire/pull/103#issuecomment-145492582 > > . > – Cheers Tibor
        Hide
        ASF GitHub Bot added a comment -

        Github user winterbe commented on the pull request:

        https://github.com/apache/maven-surefire/pull/103#issuecomment-145505027

        Great, thanks for clarifying! :+1:

        Show
        ASF GitHub Bot added a comment - Github user winterbe commented on the pull request: https://github.com/apache/maven-surefire/pull/103#issuecomment-145505027 Great, thanks for clarifying! :+1:
        Hide
        ASF GitHub Bot added a comment -

        Github user Tibor17 commented on the pull request:

        https://github.com/apache/maven-surefire/pull/103#issuecomment-145510139

        I will start the release maybe today or tomorrow. All depends on my spare
        time.

        On Mon, Oct 5, 2015 at 1:54 PM, Benjamin Winterberg <
        notifications@github.com> wrote:

        > Great, thanks for clarifying! [image: :+1:]
        >
        > —
        > Reply to this email directly or view it on GitHub
        > <https://github.com/apache/maven-surefire/pull/103#issuecomment-145505027>
        > .
        >


        Cheers
        Tibor

        Show
        ASF GitHub Bot added a comment - Github user Tibor17 commented on the pull request: https://github.com/apache/maven-surefire/pull/103#issuecomment-145510139 I will start the release maybe today or tomorrow. All depends on my spare time. On Mon, Oct 5, 2015 at 1:54 PM, Benjamin Winterberg < notifications@github.com> wrote: > Great, thanks for clarifying! [image: :+1:] > > — > Reply to this email directly or view it on GitHub > < https://github.com/apache/maven-surefire/pull/103#issuecomment-145505027 > > . > – Cheers Tibor
        Hide
        ASF GitHub Bot added a comment -

        Github user Tibor17 commented on the pull request:

        https://github.com/apache/maven-surefire/pull/103#issuecomment-145532182

        Benjamin, do you expect slight changes, let me know.
        Can you please declare Failsafe Version 2.19-SNAPSHOT and run your tests.
        Let me know your impression.

        On Mon, Oct 5, 2015 at 12:43 PM, Benjamin Winterberg <
        notifications@github.com> wrote:

        > Will skipAfterFailureCount also be available in failsafe-plugin or only
        > for surefire-plugin?
        >
        > —
        > Reply to this email directly or view it on GitHub
        > <https://github.com/apache/maven-surefire/pull/103#issuecomment-145492582>
        > .
        >


        Cheers
        Tibor

        Show
        ASF GitHub Bot added a comment - Github user Tibor17 commented on the pull request: https://github.com/apache/maven-surefire/pull/103#issuecomment-145532182 Benjamin, do you expect slight changes, let me know. Can you please declare Failsafe Version 2.19-SNAPSHOT and run your tests. Let me know your impression. On Mon, Oct 5, 2015 at 12:43 PM, Benjamin Winterberg < notifications@github.com> wrote: > Will skipAfterFailureCount also be available in failsafe-plugin or only > for surefire-plugin? > > — > Reply to this email directly or view it on GitHub > < https://github.com/apache/maven-surefire/pull/103#issuecomment-145492582 > > . > – Cheers Tibor
        Hide
        ASF GitHub Bot added a comment -

        Github user winterbe commented on the pull request:

        https://github.com/apache/maven-surefire/pull/103#issuecomment-145549204

        2.19-SNAPSHOT is not on central maven repo. Please give me a hint what repo I have to use to get the SNAPSHOT version of failsafe plugin, then I'll test it on my project.

        Show
        ASF GitHub Bot added a comment - Github user winterbe commented on the pull request: https://github.com/apache/maven-surefire/pull/103#issuecomment-145549204 2.19-SNAPSHOT is not on central maven repo. Please give me a hint what repo I have to use to get the SNAPSHOT version of failsafe plugin, then I'll test it on my project.
        Hide
        ASF GitHub Bot added a comment -

        Github user Tibor17 commented on the pull request:

        https://github.com/apache/maven-surefire/pull/103#issuecomment-145563402

        Usually it works for me. I am not behind proxy and use default settings.xml.

        On Mon, Oct 5, 2015 at 4:38 PM, Benjamin Winterberg <
        notifications@github.com> wrote:

        > 2.19-SNAPSHOT is not on central maven repo. Please give me a hint what
        > repo I have to use to get the SNAPSHOT version of failsafe plugin, then
        > I'll test it on my project.
        >
        > —
        > Reply to this email directly or view it on GitHub
        > <https://github.com/apache/maven-surefire/pull/103#issuecomment-145549204>
        > .
        >


        Cheers
        Tibor

        Show
        ASF GitHub Bot added a comment - Github user Tibor17 commented on the pull request: https://github.com/apache/maven-surefire/pull/103#issuecomment-145563402 Usually it works for me. I am not behind proxy and use default settings.xml. On Mon, Oct 5, 2015 at 4:38 PM, Benjamin Winterberg < notifications@github.com> wrote: > 2.19-SNAPSHOT is not on central maven repo. Please give me a hint what > repo I have to use to get the SNAPSHOT version of failsafe plugin, then > I'll test it on my project. > > — > Reply to this email directly or view it on GitHub > < https://github.com/apache/maven-surefire/pull/103#issuecomment-145549204 > > . > – Cheers Tibor
        Hide
        ASF GitHub Bot added a comment -

        Github user Tibor17 commented on the pull request:

        https://github.com/apache/maven-surefire/pull/103#issuecomment-145583437

        Maybe this helps
        http://stackoverflow.com/questions/7713996/is-there-a-way-to-make-maven-download-snapshot-versions-automatically

        On Mon, Oct 5, 2015 at 5:08 PM, Tibor Digana <tibor.digana@googlemail.com>
        wrote:

        > Usually it works for me. I am not behind proxy and use default
        > settings.xml.
        >
        >
        > On Mon, Oct 5, 2015 at 4:38 PM, Benjamin Winterberg <
        > notifications@github.com> wrote:
        >
        >> 2.19-SNAPSHOT is not on central maven repo. Please give me a hint what
        >> repo I have to use to get the SNAPSHOT version of failsafe plugin, then
        >> I'll test it on my project.
        >>
        >> —
        >> Reply to this email directly or view it on GitHub
        >> <https://github.com/apache/maven-surefire/pull/103#issuecomment-145549204>
        >> .
        >>
        >
        >
        >
        > –
        > Cheers
        > Tibor
        >


        Cheers
        Tibor

        Show
        ASF GitHub Bot added a comment - Github user Tibor17 commented on the pull request: https://github.com/apache/maven-surefire/pull/103#issuecomment-145583437 Maybe this helps http://stackoverflow.com/questions/7713996/is-there-a-way-to-make-maven-download-snapshot-versions-automatically On Mon, Oct 5, 2015 at 5:08 PM, Tibor Digana <tibor.digana@googlemail.com> wrote: > Usually it works for me. I am not behind proxy and use default > settings.xml. > > > On Mon, Oct 5, 2015 at 4:38 PM, Benjamin Winterberg < > notifications@github.com> wrote: > >> 2.19-SNAPSHOT is not on central maven repo. Please give me a hint what >> repo I have to use to get the SNAPSHOT version of failsafe plugin, then >> I'll test it on my project. >> >> — >> Reply to this email directly or view it on GitHub >> < https://github.com/apache/maven-surefire/pull/103#issuecomment-145549204 > >> . >> > > > > – > Cheers > Tibor > – Cheers Tibor
        Hide
        ASF GitHub Bot added a comment -

        Github user Tibor17 commented on the pull request:

        https://github.com/apache/maven-surefire/pull/103#issuecomment-145584123

        If it still does not help try with this URL
        https://repository.apache.org/content/groups/snapshots

        On Mon, Oct 5, 2015 at 4:38 PM, Benjamin Winterberg <
        notifications@github.com> wrote:

        > 2.19-SNAPSHOT is not on central maven repo. Please give me a hint what
        > repo I have to use to get the SNAPSHOT version of failsafe plugin, then
        > I'll test it on my project.
        >
        > —
        > Reply to this email directly or view it on GitHub
        > <https://github.com/apache/maven-surefire/pull/103#issuecomment-145549204>
        > .
        >


        Cheers
        Tibor

        Show
        ASF GitHub Bot added a comment - Github user Tibor17 commented on the pull request: https://github.com/apache/maven-surefire/pull/103#issuecomment-145584123 If it still does not help try with this URL https://repository.apache.org/content/groups/snapshots On Mon, Oct 5, 2015 at 4:38 PM, Benjamin Winterberg < notifications@github.com> wrote: > 2.19-SNAPSHOT is not on central maven repo. Please give me a hint what > repo I have to use to get the SNAPSHOT version of failsafe plugin, then > I'll test it on my project. > > — > Reply to this email directly or view it on GitHub > < https://github.com/apache/maven-surefire/pull/103#issuecomment-145549204 > > . > – Cheers Tibor
        Hide
        ASF GitHub Bot added a comment -

        Github user Tibor17 commented on the pull request:

        https://github.com/apache/maven-surefire/pull/103#issuecomment-145664550

        This feature has limitations own limitations, see the page after successful
        release
        https://maven.apache.org/surefire/maven-surefire-plugin/examples/skip-after-failure.html

        On Mon, Oct 5, 2015 at 6:08 PM, Tibor Digana <tibor.digana@googlemail.com>
        wrote:

        > If it still does not help try with this URL
        > https://repository.apache.org/content/groups/snapshots
        >
        > On Mon, Oct 5, 2015 at 4:38 PM, Benjamin Winterberg <
        > notifications@github.com> wrote:
        >
        >> 2.19-SNAPSHOT is not on central maven repo. Please give me a hint what
        >> repo I have to use to get the SNAPSHOT version of failsafe plugin, then
        >> I'll test it on my project.
        >>
        >> —
        >> Reply to this email directly or view it on GitHub
        >> <https://github.com/apache/maven-surefire/pull/103#issuecomment-145549204>
        >> .
        >>
        >
        >
        >
        > –
        > Cheers
        > Tibor
        >


        Cheers
        Tibor

        Show
        ASF GitHub Bot added a comment - Github user Tibor17 commented on the pull request: https://github.com/apache/maven-surefire/pull/103#issuecomment-145664550 This feature has limitations own limitations, see the page after successful release https://maven.apache.org/surefire/maven-surefire-plugin/examples/skip-after-failure.html On Mon, Oct 5, 2015 at 6:08 PM, Tibor Digana <tibor.digana@googlemail.com> wrote: > If it still does not help try with this URL > https://repository.apache.org/content/groups/snapshots > > On Mon, Oct 5, 2015 at 4:38 PM, Benjamin Winterberg < > notifications@github.com> wrote: > >> 2.19-SNAPSHOT is not on central maven repo. Please give me a hint what >> repo I have to use to get the SNAPSHOT version of failsafe plugin, then >> I'll test it on my project. >> >> — >> Reply to this email directly or view it on GitHub >> < https://github.com/apache/maven-surefire/pull/103#issuecomment-145549204 > >> . >> > > > > – > Cheers > Tibor > – Cheers Tibor
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Closed Closed
        2126d 20h 33m 1 Tibor Digana 06/Sep/15 21:03

          People

          • Assignee:
            Tibor Digana
            Reporter:
            Paul Curren
          • Votes:
            22 Vote for this issue
            Watchers:
            17 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development