Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.10, 6.0
    • Component/s: general/build, general/test
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      On dev@lao we discussed about integrating a test beaster directly into google.

      This extra target in common-build.xml does the same like Mike's Python script using Groovy.

      1. LUCENE-5881.patch
        3 kB
        Uwe Schindler
      2. LUCENE-5881.patch
        2 kB
        Uwe Schindler
      3. LUCENE-5881.patch
        2 kB
        Uwe Schindler

        Issue Links

          Activity

          Hide
          Uwe Schindler added a comment -

          From mailing list:

          Hi,

          attached you will find the beaster:

          • Only modifies common-build.xml, so no inherit down (makes no sense otherwise, as you would never run "ant beast-test" from top-level. So you have to go to correct submodule and run "ant beast-test -Dbeast.iters=n -Dtestcase=..." from there
          • Uses "antcall" in a loop, invoking the internal dependency-less "-test" target. My first impl used the test-macro directly, but this did not work, because test-macro sets non-local properties, which are then available on second round, causing errors or use always same seed. Antcall creates a new project each time and runs tests.

          I can open an issue or just commit this

          Uwe


          Uwe Schindler
          H.-H.-Meier-Allee 63, D-28213 Bremen
          http://www.thetaphi.de
          eMail: uwe@thetaphi.de

          > ----Original Message----
          > From: Uwe Schindler uwe@thetaphi.de
          > Sent: Friday, August 08, 2014 8:13 PM
          > To: dev@lucene.apache.org
          > Subject: RE: Test iterations
          >
          > Hi,
          >
          > I will look into that as a Groovy Skript: The main problem is: You
          > cannot simply use <antcall/> in a loop, because this would also
          > execute the dependencies on each run.
          >
          > My idea is to do the following:
          > - maybe subclass antcall Task with Groovy (not sure if this is needed)
          > - instantiate it with current project
          > - execute dependent targets
          > - execute the inner target multiple times: store the project
          > properties first and restore them after execution. This is done,
          > because ANT properties can only be set once. If you don't give a
          > fixed test seed, each run would pick a new one (because the project
          > properties are reset, so the seed from the previous execution is gone).
          >
          > Uwe
          >
          > -----
          > Uwe Schindler
          > H.-H.-Meier-Allee 63, D-28213 Bremen
          > http://www.thetaphi.de
          > eMail: uwe@thetaphi.de
          >
          >
          > > ----Original Message----
          > > From: Ryan Ernst ryan@iernst.net
          > > Sent: Friday, August 08, 2014 5:08 PM
          > > To: dev@lucene.apache.org
          > > Subject: Re: Test iterations
          > >
          > > Thanks for the extremely thorough answer, Dawid! Entertaining as
          > > always.
          > >
          > > > Should we provide this "beaster" in common-build?
          > >
          > > I would use it! It sounds like there is a lot of work involved in
          > > making tests.iters work better with LuceneTestCase. In the mean
          > > time, this sounds like a quick solution that might not be as
          > > efficient (multiple JVMs), but still better than having to come up with a bash script?
          > >
          > > On Fri, Aug 8, 2014 at 7:28 AM, Michael McCandless
          > > <lucene@mikemccandless.com> wrote:
          > > > +1, this sounds awesome?
          > > >
          > > > Mike McCandless
          > > >
          > > > http://blog.mikemccandless.com
          > > >
          > > >
          > > > On Fri, Aug 8, 2014 at 10:06 AM, Uwe Schindler <uwe@thetaphi.de>
          > wrote:
          > > >> Hi,
          > > >>
          > > >> We could emulate the same thing (the repeating beaster) with pure
          > Ant:
          > > >>
          > > >> Just repeat the "test" target, which can be done using ant-contrib's "for"
          > > task or (much simplier) a groovy script using antcall on the test target.
          > > >> Should we provide this "beaster" in common-build?
          > > >>
          > > >> "ant beast-tests -Dbeast.iter=100 -Dtestcase=..."
          > > >>
          > > >> Very easy to implement and makes it easier to use for the python
          > > >> haters -
          > > and comes embedded...
          > > >>
          > > >> Uwe
          > > >>
          > > >> -----
          > > >> Uwe Schindler
          > > >> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
          > > >> eMail: uwe@thetaphi.de
          > > >>
          > > >>
          > > >>> ----Original Message----
          > > >>> From: Michael McCandless lucene@mikemccandless.com
          > > >>> Sent: Friday, August 08, 2014 3:48 PM
          > > >>> To: Lucene/Solr dev
          > > >>> Subject: Re: Test iterations
          > > >>>
          > > >>> On Fri, Aug 8, 2014 at 9:35 AM, Uwe Schindler <uwe@thetaphi.de>
          > > wrote:
          > > >>> > Hi Dawid,
          > > >>> >
          > > >>> > Thanks for the very good explanation! Indeed the main problem
          > > >>> > with
          > > >>> tests.iters is the static initializers. Maybe put that
          > > >>> explanation into the Wiki! I sometimes also need to remember it,
          > > >>> so it should be
          > > documented.
          > > >>> >
          > > >>> > One (only theoretical) way to solve the whole thing could be:
          > > >>> > Load the class(es) in a separate classloader for every
          > > >>> > repeated execution,... but of course this will very fast blow
          > > >>> > up your permgen (java 6, 7) or anything else we don't know about (java 8).
          > > >>> > In fact the separate classloader approach is not different
          > > >>> > from Mike's scripts, just that Mike's script creates a new
          > > >>> > classloader by forking a new JVM. In fact I don't think the
          > > >>> > separate classloader approach would be much faster, because
          > > >>> > the class clones will all have separate compilation paths in
          > > >>> > Hotspot, so Hotspot cannot share the same assembler code. So
          > > >>> > except the JVM startup time, you gain nothing. Just permgen
          > > >>> > issues
          > > >>>
          > > >>> The big thing the python beasting scripts avoids is all the ant
          > > >>> overhead to just get to the point where it actually spawns the
          > > >>> JVM to run the test. Really, that's all the beasting script does:
          > > >>> directly spawn the JVM on the test runner (after running "ant
          > > >>> test-compile" up
          > > >>> front) and then parse its output/events.
          > > >>>
          > > >>> The distributed test runner, which uses rsync/ssh to run tests
          > > >>> on N machines, is very different from the beasting script: it
          > > >>> runs all Lucene's tests (instead of a single test over and over)
          > > >>> across N JVMs on M machines. It "cheats" by taking the union of
          > > >>> all
          > CLASSPATHs
          > > ...
          > > >>> but this is a huge win because it means all testing is fully
          > > >>> concurrent, not just concurrent within one module. This script
          > > >>> can also repeat, which means once all lucene tests finish,
          > > >>> re-en-queue all
          > of
          > > them again.

          Show
          Uwe Schindler added a comment - From mailing list: Hi, attached you will find the beaster: Only modifies common-build.xml, so no inherit down (makes no sense otherwise, as you would never run "ant beast-test" from top-level. So you have to go to correct submodule and run "ant beast-test -Dbeast.iters=n -Dtestcase=..." from there Uses "antcall" in a loop, invoking the internal dependency-less "-test" target. My first impl used the test-macro directly, but this did not work, because test-macro sets non-local properties, which are then available on second round, causing errors or use always same seed. Antcall creates a new project each time and runs tests. I can open an issue or just commit this Uwe Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: uwe@thetaphi.de > ---- Original Message ---- > From: Uwe Schindler uwe@thetaphi.de > Sent: Friday, August 08, 2014 8:13 PM > To: dev@lucene.apache.org > Subject: RE: Test iterations > > Hi, > > I will look into that as a Groovy Skript: The main problem is: You > cannot simply use <antcall/> in a loop, because this would also > execute the dependencies on each run. > > My idea is to do the following: > - maybe subclass antcall Task with Groovy (not sure if this is needed) > - instantiate it with current project > - execute dependent targets > - execute the inner target multiple times: store the project > properties first and restore them after execution. This is done, > because ANT properties can only be set once . If you don't give a > fixed test seed, each run would pick a new one (because the project > properties are reset, so the seed from the previous execution is gone). > > Uwe > > ----- > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: uwe@thetaphi.de > > > > ---- Original Message ---- > > From: Ryan Ernst ryan@iernst.net > > Sent: Friday, August 08, 2014 5:08 PM > > To: dev@lucene.apache.org > > Subject: Re: Test iterations > > > > Thanks for the extremely thorough answer, Dawid! Entertaining as > > always. > > > > > Should we provide this "beaster" in common-build? > > > > I would use it! It sounds like there is a lot of work involved in > > making tests.iters work better with LuceneTestCase. In the mean > > time, this sounds like a quick solution that might not be as > > efficient (multiple JVMs), but still better than having to come up with a bash script? > > > > On Fri, Aug 8, 2014 at 7:28 AM, Michael McCandless > > <lucene@mikemccandless.com> wrote: > > > +1, this sounds awesome? > > > > > > Mike McCandless > > > > > > http://blog.mikemccandless.com > > > > > > > > > On Fri, Aug 8, 2014 at 10:06 AM, Uwe Schindler <uwe@thetaphi.de> > wrote: > > >> Hi, > > >> > > >> We could emulate the same thing (the repeating beaster) with pure > Ant: > > >> > > >> Just repeat the "test" target, which can be done using ant-contrib's "for" > > task or (much simplier) a groovy script using antcall on the test target. > > >> Should we provide this "beaster" in common-build? > > >> > > >> "ant beast-tests -Dbeast.iter=100 -Dtestcase=..." > > >> > > >> Very easy to implement and makes it easier to use for the python > > >> haters - > > and comes embedded... > > >> > > >> Uwe > > >> > > >> ----- > > >> Uwe Schindler > > >> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de > > >> eMail: uwe@thetaphi.de > > >> > > >> > > >>> ---- Original Message ---- > > >>> From: Michael McCandless lucene@mikemccandless.com > > >>> Sent: Friday, August 08, 2014 3:48 PM > > >>> To: Lucene/Solr dev > > >>> Subject: Re: Test iterations > > >>> > > >>> On Fri, Aug 8, 2014 at 9:35 AM, Uwe Schindler <uwe@thetaphi.de> > > wrote: > > >>> > Hi Dawid, > > >>> > > > >>> > Thanks for the very good explanation! Indeed the main problem > > >>> > with > > >>> tests.iters is the static initializers. Maybe put that > > >>> explanation into the Wiki! I sometimes also need to remember it, > > >>> so it should be > > documented. > > >>> > > > >>> > One (only theoretical) way to solve the whole thing could be: > > >>> > Load the class(es) in a separate classloader for every > > >>> > repeated execution,... but of course this will very fast blow > > >>> > up your permgen (java 6, 7) or anything else we don't know about (java 8). > > >>> > In fact the separate classloader approach is not different > > >>> > from Mike's scripts, just that Mike's script creates a new > > >>> > classloader by forking a new JVM. In fact I don't think the > > >>> > separate classloader approach would be much faster, because > > >>> > the class clones will all have separate compilation paths in > > >>> > Hotspot, so Hotspot cannot share the same assembler code. So > > >>> > except the JVM startup time, you gain nothing. Just permgen > > >>> > issues > > >>> > > >>> The big thing the python beasting scripts avoids is all the ant > > >>> overhead to just get to the point where it actually spawns the > > >>> JVM to run the test. Really, that's all the beasting script does: > > >>> directly spawn the JVM on the test runner (after running "ant > > >>> test-compile" up > > >>> front) and then parse its output/events. > > >>> > > >>> The distributed test runner, which uses rsync/ssh to run tests > > >>> on N machines, is very different from the beasting script: it > > >>> runs all Lucene's tests (instead of a single test over and over) > > >>> across N JVMs on M machines. It "cheats" by taking the union of > > >>> all > CLASSPATHs > > ... > > >>> but this is a huge win because it means all testing is fully > > >>> concurrent, not just concurrent within one module. This script > > >>> can also repeat, which means once all lucene tests finish, > > >>> re-en-queue all > of > > them again.
          Hide
          Uwe Schindler added a comment -

          Patch.

          This one is based on that from mailing list, but improved:

          • more Groovy-like
          • only one instance of "antcall", executed in a loop
          Show
          Uwe Schindler added a comment - Patch. This one is based on that from mailing list, but improved: more Groovy-like only one instance of "antcall", executed in a loop
          Hide
          Robert Muir added a comment -

          +1!!! Can we just name it ant beast?

          Show
          Robert Muir added a comment - +1!!! Can we just name it ant beast?
          Hide
          Uwe Schindler added a comment -

          I can rename it before committing!

          Show
          Uwe Schindler added a comment - I can rename it before committing!
          Hide
          Uwe Schindler added a comment -

          Patch with renamed target.

          Show
          Uwe Schindler added a comment - Patch with renamed target.
          Hide
          Uwe Schindler added a comment -

          I also added docs to "test-help".

          Will commit this soon!

          Show
          Uwe Schindler added a comment - I also added docs to "test-help". Will commit this soon!
          Hide
          ASF subversion and git services added a comment -

          Commit 1617078 from Uwe Schindler in branch 'dev/trunk'
          [ https://svn.apache.org/r1617078 ]

          LUCENE-5881: Add "beasting" of tests: repeats the whole "test" Ant target N times with "ant beast -Dbeast.iters=N"

          Show
          ASF subversion and git services added a comment - Commit 1617078 from Uwe Schindler in branch 'dev/trunk' [ https://svn.apache.org/r1617078 ] LUCENE-5881 : Add "beasting" of tests: repeats the whole "test" Ant target N times with "ant beast -Dbeast.iters=N"
          Hide
          ASF subversion and git services added a comment -

          Commit 1617079 from Uwe Schindler in branch 'dev/trunk'
          [ https://svn.apache.org/r1617079 ]

          LUCENE-5881: Add parameter check, beautify some other groovy code

          Show
          ASF subversion and git services added a comment - Commit 1617079 from Uwe Schindler in branch 'dev/trunk' [ https://svn.apache.org/r1617079 ] LUCENE-5881 : Add parameter check, beautify some other groovy code
          Hide
          ASF subversion and git services added a comment -

          Commit 1617080 from Uwe Schindler in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1617080 ]

          Merged revision(s) 1617078-1617079 from lucene/dev/trunk:
          LUCENE-5881: Add "beasting" of tests: repeats the whole "test" Ant target N times with "ant beast -Dbeast.iters=N"
          LUCENE-5881: Add parameter check, beautify some other groovy code

          Show
          ASF subversion and git services added a comment - Commit 1617080 from Uwe Schindler in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1617080 ] Merged revision(s) 1617078-1617079 from lucene/dev/trunk: LUCENE-5881 : Add "beasting" of tests: repeats the whole "test" Ant target N times with "ant beast -Dbeast.iters=N" LUCENE-5881 : Add parameter check, beautify some other groovy code
          Hide
          Uwe Schindler added a comment -

          Committed to trunk an branch_4x.

          Show
          Uwe Schindler added a comment - Committed to trunk an branch_4x.
          Hide
          Dawid Weiss added a comment -

          Thanks Uwe, I couldn't comment on the patch earlier, it looks good!

          Show
          Dawid Weiss added a comment - Thanks Uwe, I couldn't comment on the patch earlier, it looks good!
          Hide
          Dawid Weiss added a comment -

          This is the original 'run tests with multiple master seeds' issue. I'll leave it open; perhaps I'll be able to implement built-in master seed repeating one day (this would allow running it in the IDE, among other benefits).

          Show
          Dawid Weiss added a comment - This is the original 'run tests with multiple master seeds' issue. I'll leave it open; perhaps I'll be able to implement built-in master seed repeating one day (this would allow running it in the IDE, among other benefits).
          Hide
          Michael McCandless added a comment -

          This is awesome, thanks Uwe!

          Show
          Michael McCandless added a comment - This is awesome, thanks Uwe!

            People

            • Assignee:
              Uwe Schindler
              Reporter:
              Uwe Schindler
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development