Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10023

Improve single unit test run time with ant.

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.5, 7.0
    • Component/s: Tests
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      It seems to take 2 minutes and 45 seconds to run a single test with the latest build design and the test itself is only 4 seconds. I've noticed this for a long time, and it seems because ant is running through a billion targets first.

      I haven't checked yet, so maybe it's a Solr specific issue? I'll check with Lucene and move this issue if necessary.

      There is hopefully something we can do to improve this though. At least we should try and get some sharp minds to take first / second look. If I did not use an IDE so much to run tests, this would drive me nuts.

      1. SOLR-10023-add-test-nocompile-target.patch
        4 kB
        Steve Rowe
      2. SOLR-10023-add-test-only-target.patch
        4 kB
        Steve Rowe
      3. SOLR-10023-add-test-only-target.patch
        4 kB
        Steve Rowe
      4. stdout.tar.gz
        720 kB
        Mark Miller

        Activity

        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        +1 to solving this. This is a huge pain point.

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - +1 to solving this. This is a huge pain point.
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        Steve Rowe, I cannot remember who was involved in a lot of the latest build system design, but for some reason your name comes to mind as someone involved. If so, do you recall this issue and if any attempt to improve on it was made?

        Show
        markrmiller@gmail.com Mark Miller added a comment - Steve Rowe , I cannot remember who was involved in a lot of the latest build system design, but for some reason your name comes to mind as someone involved. If so, do you recall this issue and if any attempt to improve on it was made?
        Hide
        dsmiley David Smiley added a comment -

        +1 yes, it takes way too long right now.

        Show
        dsmiley David Smiley added a comment - +1 yes, it takes way too long right now.
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        It's also interesting that the bulk of the time appears to be after the test has finished, and only if it's successful. Must be some way to speed this up.

        Show
        markrmiller@gmail.com Mark Miller added a comment - It's also interesting that the bulk of the time appears to be after the test has finished, and only if it's successful. Must be some way to speed this up.
        Hide
        dweiss Dawid Weiss added a comment -

        Mark: try checking which task is taking so much time.
        https://ant.apache.org/manual/listeners.html#ProfileLogger

        Looks like it could be trying to load/ parse all the test classes, even if only one of them is executed? Shouldn't be the case if it's input fileset filtering for ant though, odd.

        Show
        dweiss Dawid Weiss added a comment - Mark: try checking which task is taking so much time. https://ant.apache.org/manual/listeners.html#ProfileLogger Looks like it could be trying to load/ parse all the test classes, even if only one of them is executed? Shouldn't be the case if it's input fileset filtering for ant though, odd.
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        Thanks Dawid Weiss, attaching sample output.

        Looks like fast targets, but a lot of them over and over and over.

        Show
        markrmiller@gmail.com Mark Miller added a comment - Thanks Dawid Weiss , attaching sample output. Looks like fast targets, but a lot of them over and over and over.
        Hide
        dweiss Dawid Weiss added a comment -

        Trimmed stuff a bit with:

        grep ": finished" stdout | grep "^property" -v | grep -v "0ms" | grep "Target" 
        

        Looks like the recursion is doing things over and over again.

        Target compile-morphlines-core: finished Mon Jan 23 08:14:53 EST 2017 (3473ms) 
        Target compile-morphlines-core: finished Mon Jan 23 08:15:04 EST 2017 (7737ms) 
        Target compile-morphlines-core: finished Mon Jan 23 08:15:22 EST 2017 (9247ms) 
        

        and everything adds up eventually (jar resolving, compilations, etc.). Unfortunately ANT is very dumb at multi-module dependencies so I don't even know where to begin fixing this... Maybe we should reopen LUCENE-5755?

        Show
        dweiss Dawid Weiss added a comment - Trimmed stuff a bit with: grep ": finished" stdout | grep "^property" -v | grep -v "0ms" | grep "Target" Looks like the recursion is doing things over and over again. Target compile-morphlines-core: finished Mon Jan 23 08:14:53 EST 2017 (3473ms) Target compile-morphlines-core: finished Mon Jan 23 08:15:04 EST 2017 (7737ms) Target compile-morphlines-core: finished Mon Jan 23 08:15:22 EST 2017 (9247ms) and everything adds up eventually (jar resolving, compilations, etc.). Unfortunately ANT is very dumb at multi-module dependencies so I don't even know where to begin fixing this... Maybe we should reopen LUCENE-5755 ?
        Hide
        dweiss Dawid Weiss added a comment -

        Btw. some of the targets execute a number of times in that log file...

        Target compile-solr-core: finished Mon Jan 23 08:13:04 EST 2017 (5309ms)
        Target compile-solr-core: finished Mon Jan 23 08:13:09 EST 2017 (2714ms)
        Target compile-solr-core: finished Mon Jan 23 08:13:16 EST 2017 (3144ms)
        Target compile-solr-core: finished Mon Jan 23 08:13:58 EST 2017 (2223ms)
        Target compile-solr-core: finished Mon Jan 23 08:13:58 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:04 EST 2017 (2719ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:05 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:10 EST 2017 (2604ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:10 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:15 EST 2017 (2404ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:16 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:21 EST 2017 (2578ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:21 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:23 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:28 EST 2017 (2058ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:28 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:33 EST 2017 (2196ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:33 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:37 EST 2017 (1767ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:38 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:42 EST 2017 (1950ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:43 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:46 EST 2017 (1827ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:48 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:50 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:52 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:54 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:56 EST 2017 (1ms)
        Target compile-solr-core: finished Mon Jan 23 08:14:58 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:00 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:01 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:04 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:05 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:09 EST 2017 (2467ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:11 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:13 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:15 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:18 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:19 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:22 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:27 EST 2017 (2849ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:28 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:29 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:35 EST 2017 (2393ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:35 EST 2017 (0ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:40 EST 2017 (2081ms)
        Target compile-solr-core: finished Mon Jan 23 08:15:40 EST 2017 (0ms) 
        
        Show
        dweiss Dawid Weiss added a comment - Btw. some of the targets execute a number of times in that log file... Target compile-solr-core: finished Mon Jan 23 08:13:04 EST 2017 (5309ms) Target compile-solr-core: finished Mon Jan 23 08:13:09 EST 2017 (2714ms) Target compile-solr-core: finished Mon Jan 23 08:13:16 EST 2017 (3144ms) Target compile-solr-core: finished Mon Jan 23 08:13:58 EST 2017 (2223ms) Target compile-solr-core: finished Mon Jan 23 08:13:58 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:14:04 EST 2017 (2719ms) Target compile-solr-core: finished Mon Jan 23 08:14:05 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:14:10 EST 2017 (2604ms) Target compile-solr-core: finished Mon Jan 23 08:14:10 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:14:15 EST 2017 (2404ms) Target compile-solr-core: finished Mon Jan 23 08:14:16 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:14:21 EST 2017 (2578ms) Target compile-solr-core: finished Mon Jan 23 08:14:21 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:14:23 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:14:28 EST 2017 (2058ms) Target compile-solr-core: finished Mon Jan 23 08:14:28 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:14:33 EST 2017 (2196ms) Target compile-solr-core: finished Mon Jan 23 08:14:33 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:14:37 EST 2017 (1767ms) Target compile-solr-core: finished Mon Jan 23 08:14:38 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:14:42 EST 2017 (1950ms) Target compile-solr-core: finished Mon Jan 23 08:14:43 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:14:46 EST 2017 (1827ms) Target compile-solr-core: finished Mon Jan 23 08:14:48 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:14:50 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:14:52 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:14:54 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:14:56 EST 2017 (1ms) Target compile-solr-core: finished Mon Jan 23 08:14:58 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:15:00 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:15:01 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:15:04 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:15:05 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:15:09 EST 2017 (2467ms) Target compile-solr-core: finished Mon Jan 23 08:15:11 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:15:13 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:15:15 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:15:18 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:15:19 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:15:22 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:15:27 EST 2017 (2849ms) Target compile-solr-core: finished Mon Jan 23 08:15:28 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:15:29 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:15:35 EST 2017 (2393ms) Target compile-solr-core: finished Mon Jan 23 08:15:35 EST 2017 (0ms) Target compile-solr-core: finished Mon Jan 23 08:15:40 EST 2017 (2081ms) Target compile-solr-core: finished Mon Jan 23 08:15:40 EST 2017 (0ms)
        Hide
        steve_rowe Steve Rowe added a comment -

        Initial first thought: If you want speed, you should run individual tests from the module that contains them. Otherwise, the build will update compilation and look for matching tests in every module at and below where you are. I think that's what your log output shows Mark Miller? And that will of course be slow.

        Show
        steve_rowe Steve Rowe added a comment - Initial first thought: If you want speed, you should run individual tests from the module that contains them. Otherwise, the build will update compilation and look for matching tests in every module at and below where you are. I think that's what your log output shows Mark Miller ? And that will of course be slow.
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        If you want speed, you should run individual tests from the module that contains them

        We should almost fail single test running from a higher level and tell the dev to do this then - would not have guessed it would make so much difference. I'll give that a try.

        Looks like the recursion is doing things over and over again.

        Probably not so easy to fix sadly. I found that if you remove the test call depends on compile and just run the precompiled test, I could go from 2 minutes 40 seconds to about 40 seconds of build time.

        Show
        markrmiller@gmail.com Mark Miller added a comment - If you want speed, you should run individual tests from the module that contains them We should almost fail single test running from a higher level and tell the dev to do this then - would not have guessed it would make so much difference. I'll give that a try. Looks like the recursion is doing things over and over again. Probably not so easy to fix sadly. I found that if you remove the test call depends on compile and just run the precompiled test, I could go from 2 minutes 40 seconds to about 40 seconds of build time.
        Hide
        dweiss Dawid Weiss added a comment -

        I bet the timings here could be cut to very reasonable few seconds... if the dependencies are scanned properly once, not over and over again. But I don't know how to do this in Ant, Maven has its own set of nighmares (-am -pl ...) and a complex Gradle build is no simpler to understand than a complex Ant build (my personal opinion).

        Back to drawing board. Or Make...

        Show
        dweiss Dawid Weiss added a comment - I bet the timings here could be cut to very reasonable few seconds... if the dependencies are scanned properly once, not over and over again. But I don't know how to do this in Ant, Maven has its own set of nighmares ( -am -pl ... ) and a complex Gradle build is no simpler to understand than a complex Ant build (my personal opinion). Back to drawing board. Or Make ...
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        Steve is right that going into the module is light years better. In my script I at least can now regex out the module (or contrib) from the test file path and then go into the right module to run the test. Would be a cool hack if the top level solr test even just did that for you. That or it probably should fail and tell you how to run a single test rather than punish you with an extra 2 minutes plus.

        Show
        markrmiller@gmail.com Mark Miller added a comment - Steve is right that going into the module is light years better. In my script I at least can now regex out the module (or contrib) from the test file path and then go into the right module to run the test. Would be a cool hack if the top level solr test even just did that for you. That or it probably should fail and tell you how to run a single test rather than punish you with an extra 2 minutes plus.
        Hide
        steve_rowe Steve Rowe added a comment -

        We could make a module-level target test-only (or similar name) that only runs tests, and doesn't attempt to enable clover, or download jars, or update compilation.

        module-build.xml uses common-build.xml's test target:

          <target name="test" depends="clover,compile-test,install-junit4-taskdef,validate,-init-totals,-test,-check-totals" description="Runs unit tests"/>
        

        I think we could get away with just:

          <target name="test-only" depends="install-junit4-taskdef,-init-totals,-test,-check-totals" description="Only runs unit tests.  Jars are not download; compilation is not updated; and clover is not enabled."/>
        
        Show
        steve_rowe Steve Rowe added a comment - We could make a module-level target test-only (or similar name) that only runs tests, and doesn't attempt to enable clover, or download jars, or update compilation. module-build.xml uses common-build.xml 's test target: <target name= "test" depends= "clover,compile-test,install-junit4-taskdef,validate,-init-totals,-test,-check-totals" description= "Runs unit tests" /> I think we could get away with just: <target name= "test-only" depends= "install-junit4-taskdef,-init-totals,-test,-check-totals" description= "Only runs unit tests. Jars are not download; compilation is not updated; and clover is not enabled." />
        Hide
        steve_rowe Steve Rowe added a comment -

        Attaching patch that adds a non-recursive test-only target, works for me, including running single tests (with -Dtestcase=...) or all tests.

        If you run ant test-only without first running ant compile-test, then you'll get an error like:

        BUILD FAILED
        /Users/sarowe/git/lucene-solr-6/lucene/common-build.xml:1448: The following error occurred while executing this line:
        /Users/sarowe/git/lucene-solr-6/lucene/common-build.xml:985: /Users/sarowe/git/lucene-solr-6/lucene/build/core/classes/test does not exist.
        

        Running ant compile-test test-only will always avoid the above error, but is almost what ant test does, and so will operate at the same (slow) speed.

        If you run ant test-only from a non-module directory with a build.xml file (currently lucene/, solr/, and lucene/analysis/), you'll get an error like:

        BUILD FAILED
        /Users/sarowe/git/lucene-solr-6/lucene/build.xml:60: Target 'test-only' will not run recursively.  First change directory to the module you want to test.
        

        I think it's ready to go. Should we do this?

        Show
        steve_rowe Steve Rowe added a comment - Attaching patch that adds a non-recursive test-only target, works for me, including running single tests (with -Dtestcase=... ) or all tests. If you run ant test-only without first running ant compile-test , then you'll get an error like: BUILD FAILED /Users/sarowe/git/lucene-solr-6/lucene/common-build.xml:1448: The following error occurred while executing this line: /Users/sarowe/git/lucene-solr-6/lucene/common-build.xml:985: /Users/sarowe/git/lucene-solr-6/lucene/build/core/classes/test does not exist. Running ant compile-test test-only will always avoid the above error, but is almost what ant test does, and so will operate at the same (slow) speed. If you run ant test-only from a non-module directory with a build.xml file (currently lucene/ , solr/ , and lucene/analysis/ ), you'll get an error like: BUILD FAILED /Users/sarowe/git/lucene-solr-6/lucene/build.xml:60: Target 'test-only' will not run recursively. First change directory to the module you want to test. I think it's ready to go. Should we do this?
        Hide
        steve_rowe Steve Rowe added a comment -

        Modified patch to describe ant test-only in ant test-help's output.

        Show
        steve_rowe Steve Rowe added a comment - Modified patch to describe ant test-only in ant test-help 's output.
        Hide
        mdrob Mike Drob added a comment -

        I don't think skipping compile is the right way to do this. Often I'm making one change, running the test, making another change, in a tight loop. Maybe running compile+test from the module is sufficient, but I see this tripping somebody up down the road.

        Show
        mdrob Mike Drob added a comment - I don't think skipping compile is the right way to do this. Often I'm making one change, running the test, making another change, in a tight loop. Maybe running compile+test from the module is sufficient, but I see this tripping somebody up down the road.
        Hide
        yseeley@gmail.com Yonik Seeley added a comment -

        Attaching patch that adds a non-recursive test-only

        +1, I've been desperately looking for something like this! I used to maintain hacked build scripts privately, but stopped long ago due to the cost of maintenance.

        tripping somebody up down the road.

        Possibly, but one should run the full test suite before committing anything anyway. One could argue that even the ability to run a single test in isolation can trip people up (because an assumption is made about the scope of a change).

        Show
        yseeley@gmail.com Yonik Seeley added a comment - Attaching patch that adds a non-recursive test-only +1, I've been desperately looking for something like this! I used to maintain hacked build scripts privately, but stopped long ago due to the cost of maintenance. tripping somebody up down the road. Possibly, but one should run the full test suite before committing anything anyway. One could argue that even the ability to run a single test in isolation can trip people up (because an assumption is made about the scope of a change).
        Hide
        mdrob Mike Drob added a comment -

        Yea, I've been thinking about the use cases for this, and a good related change would be to fix suggest the test only goal when a unit test fails. No idea where that message comes from though: reproduce with: ant test-only ...

        Show
        mdrob Mike Drob added a comment - Yea, I've been thinking about the use cases for this, and a good related change would be to fix suggest the test only goal when a unit test fails. No idea where that message comes from though: reproduce with: ant test-only ...
        Hide
        steve_rowe Steve Rowe added a comment -

        I see this tripping somebody up down the road.

        I do too, hence "Should we do this?".

        But for the tweak-compile-test cycle you describe, you can just use ant test, right? In other words, your concern isn't lack of support for that pattern, but rather that ill-informed people would be lulled into thinking they're testing up-to-date compilations of source code rather than previous compilations. One way to address this could be to reduce advertisement of the target, e.g. removing the target's description so that it doesn't show up when a user runs ant -p.

        My opinion (and why I did the work): we should provide both options, and educate users about the difference.

        the use cases for this

        I think the main use case is for external beasting of various kinds.

        a good related change would be to fix suggest the test only goal when a unit test fails. No idea where that message comes from though: reproduce with: ant test-only ...

        I don't think this is a good idea, for the same reason you mentioned: skipping compilation. Yes, re-running a previously compiled test is the classic reproduction scenario, but a dev is IMHO just as likely to use your tweak-compile-test cycle in this case. Changing the printed repro line would serve as a form of advertisement for the feature generally, which I think is probably the wrong direction for this kind of functionality.

        Show
        steve_rowe Steve Rowe added a comment - I see this tripping somebody up down the road. I do too, hence "Should we do this?". But for the tweak-compile-test cycle you describe, you can just use ant test , right? In other words, your concern isn't lack of support for that pattern, but rather that ill-informed people would be lulled into thinking they're testing up-to-date compilations of source code rather than previous compilations. One way to address this could be to reduce advertisement of the target, e.g. removing the target's description so that it doesn't show up when a user runs ant -p . My opinion (and why I did the work): we should provide both options, and educate users about the difference. the use cases for this I think the main use case is for external beasting of various kinds. a good related change would be to fix suggest the test only goal when a unit test fails. No idea where that message comes from though: reproduce with: ant test-only ... I don't think this is a good idea, for the same reason you mentioned: skipping compilation. Yes, re-running a previously compiled test is the classic reproduction scenario, but a dev is IMHO just as likely to use your tweak-compile-test cycle in this case. Changing the printed repro line would serve as a form of advertisement for the feature generally, which I think is probably the wrong direction for this kind of functionality.
        Hide
        mdrob Mike Drob added a comment -

        I think the main use case is for external beasting of various kinds.

        Isn't that what ant beast is for? Which already refuses to run from outside of the specific module, cutting down on a lot of time there.

        Show
        mdrob Mike Drob added a comment - I think the main use case is for external beasting of various kinds. Isn't that what ant beast is for? Which already refuses to run from outside of the specific module, cutting down on a lot of time there.
        Hide
        erickerickson Erick Erickson added a comment -

        Well, let's make the command test-nocompile then. And lets advertise it in the help target. With a big fat warning maybe.

        Not providing something like this because it might inconvenience someone new at the expense of slowing down development is a bad tradeoff IMO. With an appropriate name it ought to be pretty safe.

        Right now, I made a change and have three test failures. They look like I got unlucky with some of the flaky tests so I want to run them one by one....

        Show
        erickerickson Erick Erickson added a comment - Well, let's make the command test-nocompile then. And lets advertise it in the help target. With a big fat warning maybe. Not providing something like this because it might inconvenience someone new at the expense of slowing down development is a bad tradeoff IMO. With an appropriate name it ought to be pretty safe. Right now, I made a change and have three test failures. They look like I got unlucky with some of the flaky tests so I want to run them one by one....
        Hide
        steve_rowe Steve Rowe added a comment -

        Right now, I made a change and have three test failures. They look like I got unlucky with some of the flaky tests so I want to run them one by one....

        I would like a way to say with a simple command: go re-run all tests that failed in the last run. And then I would like to use this on my Jenkins in some automatic way so that I only get failure emails for reproducing failures. There are some non-reproducing failures that are also problematic, but mostly flakey tests are going to be pretty regularly flakey. So I guess the ideal report would be some combination of a) reproducing failures and b) non-reproducing but unusual failures.

        Show
        steve_rowe Steve Rowe added a comment - Right now, I made a change and have three test failures. They look like I got unlucky with some of the flaky tests so I want to run them one by one.... I would like a way to say with a simple command: go re-run all tests that failed in the last run. And then I would like to use this on my Jenkins in some automatic way so that I only get failure emails for reproducing failures. There are some non-reproducing failures that are also problematic, but mostly flakey tests are going to be pretty regularly flakey. So I guess the ideal report would be some combination of a) reproducing failures and b) non-reproducing but unusual failures.
        Hide
        steve_rowe Steve Rowe added a comment -

        Well, let's make the command test-nocompile then.

        +1, that name sounds better to me.

        Show
        steve_rowe Steve Rowe added a comment - Well, let's make the command test-nocompile then. +1, that name sounds better to me.
        Hide
        erickerickson Erick Erickson added a comment -

        Hmmm, it's embarrassing how long I do things the hard way sometimes. Mark just commented that "changing to the module is faster". So I decided to try it. Running "ant -Dtestcase=TestLazyCores test" takes about 2.5 minutes when run from the root or root/solr. It takes 47 seconds when run from root/solr/core. You can't go any farther down than root/solr/core.
        Siiiigggggh.

        Show
        erickerickson Erick Erickson added a comment - Hmmm, it's embarrassing how long I do things the hard way sometimes. Mark just commented that "changing to the module is faster". So I decided to try it. Running "ant -Dtestcase=TestLazyCores test" takes about 2.5 minutes when run from the root or root/solr. It takes 47 seconds when run from root/solr/core. You can't go any farther down than root/solr/core. Siiiigggggh.
        Hide
        dweiss Dawid Weiss added a comment -

        Hmmm, it's embarrassing how long I do things the hard way sometimes.

        I don't think it's your fault. To me it's still a fault/ weakness of the build system – the difference will obviously always be there if you run a multi-module vs. single module run, but it should not amount to such ridiculous numbers. Not that fixing this is a simple thing to do, of course. We have Maven multi-module projects and running a specific task no a submodule has always been a pain too.

        Show
        dweiss Dawid Weiss added a comment - Hmmm, it's embarrassing how long I do things the hard way sometimes. I don't think it's your fault. To me it's still a fault/ weakness of the build system – the difference will obviously always be there if you run a multi-module vs. single module run, but it should not amount to such ridiculous numbers. Not that fixing this is a simple thing to do, of course. We have Maven multi-module projects and running a specific task no a submodule has always been a pain too.
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        +1 on this target, +1 on new name. There is still a great little speed bump for short tests to omit compile even when dropping into the module and it would be great to not have to hack it.

        Show
        markrmiller@gmail.com Mark Miller added a comment - +1 on this target, +1 on new name. There is still a great little speed bump for short tests to omit compile even when dropping into the module and it would be great to not have to hack it.
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        Thanks by way Steve Rowe.

        My beast script went from 2 minutes 40 seconds for a 4 second test to about 40 seconds dropping into module, to not much more than 4 seconds with your patch and the new test-only target. Much nicer than having to hack out compiles.

        Show
        markrmiller@gmail.com Mark Miller added a comment - Thanks by way Steve Rowe . My beast script went from 2 minutes 40 seconds for a 4 second test to about 40 seconds dropping into module, to not much more than 4 seconds with your patch and the new test-only target. Much nicer than having to hack out compiles.
        Hide
        steve_rowe Steve Rowe added a comment -

        Patch, differs from previous patch only in that the "test-only" target is renamed to "test-nocompile".

        If there are no objections, I'll commit in a day or so.

        Show
        steve_rowe Steve Rowe added a comment - Patch, differs from previous patch only in that the "test-only" target is renamed to "test-nocompile". If there are no objections, I'll commit in a day or so.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 301deb336008b4a1c54c8fc6bcfd672ea2c9a4ab in lucene-solr's branch refs/heads/branch_6x from Steve Rowe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=301deb3 ]

        SOLR-10023: Add non-recursive 'test-nocompile' target: Only runs unit tests. Jars are not download; compilation is not updated; and Clover is not enabled.

        Show
        jira-bot ASF subversion and git services added a comment - Commit 301deb336008b4a1c54c8fc6bcfd672ea2c9a4ab in lucene-solr's branch refs/heads/branch_6x from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=301deb3 ] SOLR-10023 : Add non-recursive 'test-nocompile' target: Only runs unit tests. Jars are not download; compilation is not updated; and Clover is not enabled.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 7ccbc24edfdace9ae9cea9e889e8a6c470a31012 in lucene-solr's branch refs/heads/branch_6x from Steve Rowe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7ccbc24 ]

        SOLR-10023: add CHANGES entry

        Show
        jira-bot ASF subversion and git services added a comment - Commit 7ccbc24edfdace9ae9cea9e889e8a6c470a31012 in lucene-solr's branch refs/heads/branch_6x from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7ccbc24 ] SOLR-10023 : add CHANGES entry
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 42b71955c04198ef8ff0f6208c724206042ec135 in lucene-solr's branch refs/heads/branch_6x from Steve Rowe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=42b7195 ]

        SOLR-10023: Fix typo in 'test-nocompile' target description.

        Show
        jira-bot ASF subversion and git services added a comment - Commit 42b71955c04198ef8ff0f6208c724206042ec135 in lucene-solr's branch refs/heads/branch_6x from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=42b7195 ] SOLR-10023 : Fix typo in 'test-nocompile' target description.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 2d896ee8d2c45223f6a568f7c115e3f1da9631aa in lucene-solr's branch refs/heads/master from Steve Rowe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2d896ee ]

        SOLR-10023: Add non-recursive 'test-nocompile' target: Only runs unit tests. Jars are not download; compilation is not updated; and Clover is not enabled.

        Show
        jira-bot ASF subversion and git services added a comment - Commit 2d896ee8d2c45223f6a568f7c115e3f1da9631aa in lucene-solr's branch refs/heads/master from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2d896ee ] SOLR-10023 : Add non-recursive 'test-nocompile' target: Only runs unit tests. Jars are not download; compilation is not updated; and Clover is not enabled.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 59f09f96b8286e6ea8869dd928b1b838a85c6f44 in lucene-solr's branch refs/heads/master from Steve Rowe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=59f09f9 ]

        SOLR-10023: add CHANGES entry

        Show
        jira-bot ASF subversion and git services added a comment - Commit 59f09f96b8286e6ea8869dd928b1b838a85c6f44 in lucene-solr's branch refs/heads/master from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=59f09f9 ] SOLR-10023 : add CHANGES entry
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 9fc13198ce48b678f341e976dbe45495ba41b2e4 in lucene-solr's branch refs/heads/master from Steve Rowe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9fc1319 ]

        SOLR-10023: Fix typo in 'test-nocompile' target description.

        Show
        jira-bot ASF subversion and git services added a comment - Commit 9fc13198ce48b678f341e976dbe45495ba41b2e4 in lucene-solr's branch refs/heads/master from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9fc1319 ] SOLR-10023 : Fix typo in 'test-nocompile' target description.
        Hide
        steve_rowe Steve Rowe added a comment -

        Committed the patch with the 'test-nocompile' target addition.

        I'm resolving this as fixed, which is only kinda true. If we want other kinds of cleanup, lets open new issue(s).

        Show
        steve_rowe Steve Rowe added a comment - Committed the patch with the 'test-nocompile' target addition. I'm resolving this as fixed, which is only kinda true. If we want other kinds of cleanup, lets open new issue(s).
        Hide
        steve_rowe Steve Rowe added a comment -

        I think the main use case is for external beasting of various kinds.

        Isn't that what ant beast is for? Which already refuses to run from outside of the specific module, cutting down on a lot of time there.

        Mike Drob: IIRC, ant beast is not efficient at searching for reproducing failure seeds, since parallel execution (via tests.dups) assigns the same seed to each concurrent run. Other (aka "external") beasting methods, e.g. Mark Miller's beasting script (sorry, don't have a link handy), address this weakness directly by allowing each concurrent run to select its own seed. There are other beasting methods/scripts that achieve the same end too (e.g. Bash loops).

        Show
        steve_rowe Steve Rowe added a comment - I think the main use case is for external beasting of various kinds. Isn't that what ant beast is for? Which already refuses to run from outside of the specific module, cutting down on a lot of time there. Mike Drob : IIRC, ant beast is not efficient at searching for reproducing failure seeds, since parallel execution (via tests.dups ) assigns the same seed to each concurrent run. Other (aka "external") beasting methods, e.g. Mark Miller 's beasting script (sorry, don't have a link handy), address this weakness directly by allowing each concurrent run to select its own seed. There are other beasting methods/scripts that achieve the same end too (e.g. Bash loops).
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        Right, an external script can make each run truly independent. It also allows for custom management of results and logs.

        ant beast has to make trade offs to live within the build system like it does.

        Show
        markrmiller@gmail.com Mark Miller added a comment - Right, an external script can make each run truly independent. It also allows for custom management of results and logs. ant beast has to make trade offs to live within the build system like it does.

          People

          • Assignee:
            steve_rowe Steve Rowe
            Reporter:
            markrmiller@gmail.com Mark Miller
          • Votes:
            1 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development