Lucene - Core
  1. Lucene - Core
  2. LUCENE-4360

Support running the same test suite multiple times in parallel

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0, 6.0
    • Component/s: general/test
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      The current "test execution multiplier" or:

      -Dtests.iters=N
      

      generates multiple tests (method executions) under a test class (suite). All these tests, however, are bound to a single class so they must run sequentially and on a single JVM (because of how JUnit works – nesting of rules, class hooks, etc.).

      Mark pointed out that if somebody has a multi-core CPU then it'd be nice to be able to run a single suite in parallel, possibly in combination with tests.iters (so that a single test method is executed X times on Y parallel JVMs).

      This is surprisingly easy with the randomized runner because it currently accepts "duplicate" suite names and will load-balance them in a normal way. So, if one has Y cores (JVMs) then providing a suite name X times will result in X executions, balanced across Y JVMs.

      The only problem is how to "multiply" suite names. This can be done in a number of ways, starting from a custom resource collection wrapper and ending at a built-in code in the runner itself. I think the custom collection wrapper approach would be interesting, I'll explore this direction.

        Activity

        Hide
        Dawid Weiss added a comment -

        This is a quick hack that multiplies execution of all tests by N-fold by just repeating the same fileset a few times.

        A proof of concept, but working.

        Show
        Dawid Weiss added a comment - This is a quick hack that multiplies execution of all tests by N-fold by just repeating the same fileset a few times. A proof of concept, but working.
        Hide
        Dawid Weiss added a comment -

        Any ideas what to call this parameter? It's not the same as tests.iters because it multiplies entire suites. I came up with:

        -Dtests.multiply
        -Dtests.duplicate
        
        or the short equivalents:
        
        -Dtests.mply
        -Dtests.dups
        
        Show
        Dawid Weiss added a comment - Any ideas what to call this parameter? It's not the same as tests.iters because it multiplies entire suites. I came up with: -Dtests.multiply -Dtests.duplicate or the short equivalents: -Dtests.mply -Dtests.dups
        Hide
        Dawid Weiss added a comment -

        Example of use:

        ant test-core -Dtests.dups=3 -Dtestcase=TestCharFilter
        
        <JUnit4> says kaixo! Master seed: 4CE0022F3CDFF5D7
        Your default console's encoding may not display certain unicode glyphs: windows-1252
        Executing 3 suites with 3 JVMs.
        
        Suite: org.apache.lucene.analysis.TestCharFilter
        OK      0.05s J1 | TestCharFilter.testCharFilter1
        OK      0.01s J1 | TestCharFilter.testCharFilter11
        OK      0.00s J1 | TestCharFilter.testCharFilter2
        OK      0.00s J1 | TestCharFilter.testCharFilter12
        Completed on J1 in 0.35s, 4 tests
        
        Suite: org.apache.lucene.analysis.TestCharFilter
        OK      0.04s J2 | TestCharFilter.testCharFilter1
        OK      0.01s J2 | TestCharFilter.testCharFilter11
        OK      0.00s J2 | TestCharFilter.testCharFilter2
        OK      0.01s J2 | TestCharFilter.testCharFilter12
        Completed on J2 in 0.35s, 4 tests
        
        Suite: org.apache.lucene.analysis.TestCharFilter
        OK      0.04s J0 | TestCharFilter.testCharFilter1
        OK      0.02s J0 | TestCharFilter.testCharFilter11
        OK      0.00s J0 | TestCharFilter.testCharFilter2
        OK      0.00s J0 | TestCharFilter.testCharFilter12
        Completed on J0 in 0.36s, 4 tests
        
        JVM J0:     0.67 ..     1.56 =     0.89s
        JVM J1:     0.67 ..     1.55 =     0.88s
        JVM J2:     0.67 ..     1.57 =     0.90s
        Execution time total: 1.58 sec.
        Tests summary: 3 suites, 12 tests
        

        In combination with tests.iters:

        ant test-core -Dtests.dups=2 -Dtests.iters=2 -Dtestcase=TestCharFilter
        
        <JUnit4> says íHola! Master seed: 592442048D98540B
        Your default console's encoding may not display certain unicode glyphs: windows-1252
        Executing 2 suites with 2 JVMs.
        
        Suite: org.apache.lucene.analysis.TestCharFilter
        OK      0.04s J1 | TestCharFilter.testCharFilter12 {#0 seed=[592442048D98540B:5E8780464DC70845]}
        OK      0.00s J1 | TestCharFilter.testCharFilter12 {#1 seed=[592442048D98540B:EAD13CBA7905C369]}
        OK      0.00s J1 | TestCharFilter.testCharFilter11 {#0 seed=[592442048D98540B:4AC9FEB2457325F4]}
        OK      0.00s J1 | TestCharFilter.testCharFilter11 {#1 seed=[592442048D98540B:FE9F424E71B1EED8]}
        OK      0.00s J1 | TestCharFilter.testCharFilter1 {#0 seed=[592442048D98540B:643276B2D133E783]}
        OK      0.00s J1 | TestCharFilter.testCharFilter1 {#1 seed=[592442048D98540B:D064CA4EE5F12CAF]}
        OK      0.00s J1 | TestCharFilter.testCharFilter2 {#0 seed=[592442048D98540B:D339FCA65F87D49C]}
        OK      0.01s J1 | TestCharFilter.testCharFilter2 {#1 seed=[592442048D98540B:676F405A6B451FB0]}
        Completed on J1 in 0.34s, 8 tests
        
        Suite: org.apache.lucene.analysis.TestCharFilter
        OK      0.05s J0 | TestCharFilter.testCharFilter12 {#0 seed=[592442048D98540B:5E8780464DC70845]}
        OK      0.00s J0 | TestCharFilter.testCharFilter12 {#1 seed=[592442048D98540B:EAD13CBA7905C369]}
        OK      0.00s J0 | TestCharFilter.testCharFilter11 {#0 seed=[592442048D98540B:4AC9FEB2457325F4]}
        OK      0.00s J0 | TestCharFilter.testCharFilter11 {#1 seed=[592442048D98540B:FE9F424E71B1EED8]}
        OK      0.00s J0 | TestCharFilter.testCharFilter1 {#0 seed=[592442048D98540B:643276B2D133E783]}
        OK      0.00s J0 | TestCharFilter.testCharFilter1 {#1 seed=[592442048D98540B:D064CA4EE5F12CAF]}
        OK      0.00s J0 | TestCharFilter.testCharFilter2 {#0 seed=[592442048D98540B:D339FCA65F87D49C]}
        OK      0.00s J0 | TestCharFilter.testCharFilter2 {#1 seed=[592442048D98540B:676F405A6B451FB0]}
        Completed on J0 in 0.35s, 8 tests
        
        JVM J0:     0.66 ..     1.49 =     0.82s
        JVM J1:     0.66 ..     1.28 =     0.62s
        Execution time total: 1.49 sec.
        Tests summary: 2 suites, 16 tests
        

        Note each JVM will start from the same master seed. Each test will work in its own working directory and JVM though so it can be useful for performing stress testing of a single suite.

        Show
        Dawid Weiss added a comment - Example of use: ant test-core -Dtests.dups=3 -Dtestcase=TestCharFilter <JUnit4> says kaixo! Master seed: 4CE0022F3CDFF5D7 Your default console's encoding may not display certain unicode glyphs: windows-1252 Executing 3 suites with 3 JVMs. Suite: org.apache.lucene.analysis.TestCharFilter OK 0.05s J1 | TestCharFilter.testCharFilter1 OK 0.01s J1 | TestCharFilter.testCharFilter11 OK 0.00s J1 | TestCharFilter.testCharFilter2 OK 0.00s J1 | TestCharFilter.testCharFilter12 Completed on J1 in 0.35s, 4 tests Suite: org.apache.lucene.analysis.TestCharFilter OK 0.04s J2 | TestCharFilter.testCharFilter1 OK 0.01s J2 | TestCharFilter.testCharFilter11 OK 0.00s J2 | TestCharFilter.testCharFilter2 OK 0.01s J2 | TestCharFilter.testCharFilter12 Completed on J2 in 0.35s, 4 tests Suite: org.apache.lucene.analysis.TestCharFilter OK 0.04s J0 | TestCharFilter.testCharFilter1 OK 0.02s J0 | TestCharFilter.testCharFilter11 OK 0.00s J0 | TestCharFilter.testCharFilter2 OK 0.00s J0 | TestCharFilter.testCharFilter12 Completed on J0 in 0.36s, 4 tests JVM J0: 0.67 .. 1.56 = 0.89s JVM J1: 0.67 .. 1.55 = 0.88s JVM J2: 0.67 .. 1.57 = 0.90s Execution time total: 1.58 sec. Tests summary: 3 suites, 12 tests In combination with tests.iters: ant test-core -Dtests.dups=2 -Dtests.iters=2 -Dtestcase=TestCharFilter <JUnit4> says íHola! Master seed: 592442048D98540B Your default console's encoding may not display certain unicode glyphs: windows-1252 Executing 2 suites with 2 JVMs. Suite: org.apache.lucene.analysis.TestCharFilter OK 0.04s J1 | TestCharFilter.testCharFilter12 {#0 seed=[592442048D98540B:5E8780464DC70845]} OK 0.00s J1 | TestCharFilter.testCharFilter12 {#1 seed=[592442048D98540B:EAD13CBA7905C369]} OK 0.00s J1 | TestCharFilter.testCharFilter11 {#0 seed=[592442048D98540B:4AC9FEB2457325F4]} OK 0.00s J1 | TestCharFilter.testCharFilter11 {#1 seed=[592442048D98540B:FE9F424E71B1EED8]} OK 0.00s J1 | TestCharFilter.testCharFilter1 {#0 seed=[592442048D98540B:643276B2D133E783]} OK 0.00s J1 | TestCharFilter.testCharFilter1 {#1 seed=[592442048D98540B:D064CA4EE5F12CAF]} OK 0.00s J1 | TestCharFilter.testCharFilter2 {#0 seed=[592442048D98540B:D339FCA65F87D49C]} OK 0.01s J1 | TestCharFilter.testCharFilter2 {#1 seed=[592442048D98540B:676F405A6B451FB0]} Completed on J1 in 0.34s, 8 tests Suite: org.apache.lucene.analysis.TestCharFilter OK 0.05s J0 | TestCharFilter.testCharFilter12 {#0 seed=[592442048D98540B:5E8780464DC70845]} OK 0.00s J0 | TestCharFilter.testCharFilter12 {#1 seed=[592442048D98540B:EAD13CBA7905C369]} OK 0.00s J0 | TestCharFilter.testCharFilter11 {#0 seed=[592442048D98540B:4AC9FEB2457325F4]} OK 0.00s J0 | TestCharFilter.testCharFilter11 {#1 seed=[592442048D98540B:FE9F424E71B1EED8]} OK 0.00s J0 | TestCharFilter.testCharFilter1 {#0 seed=[592442048D98540B:643276B2D133E783]} OK 0.00s J0 | TestCharFilter.testCharFilter1 {#1 seed=[592442048D98540B:D064CA4EE5F12CAF]} OK 0.00s J0 | TestCharFilter.testCharFilter2 {#0 seed=[592442048D98540B:D339FCA65F87D49C]} OK 0.00s J0 | TestCharFilter.testCharFilter2 {#1 seed=[592442048D98540B:676F405A6B451FB0]} Completed on J0 in 0.35s, 8 tests JVM J0: 0.66 .. 1.49 = 0.82s JVM J1: 0.66 .. 1.28 = 0.62s Execution time total: 1.49 sec. Tests summary: 2 suites, 16 tests Note each JVM will start from the same master seed. Each test will work in its own working directory and JVM though so it can be useful for performing stress testing of a single suite.
        Hide
        Sami Siren added a comment -

        Cool! This is a really nice addition!

        Show
        Sami Siren added a comment - Cool! This is a really nice addition!
        Hide
        Dawid Weiss added a comment -

        Thanks! It was Mark's request, actually. I had all the infrastructure ready, I just needed that duplicating resource collection and this was trivial.

        Show
        Dawid Weiss added a comment - Thanks! It was Mark's request, actually. I had all the infrastructure ready, I just needed that duplicating resource collection and this was trivial.
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Dawid Weiss
        http://svn.apache.org/viewvc?view=revision&revision=1381124

        LUCENE-4360: Support running the same test suite multiple times in parallel.

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Dawid Weiss http://svn.apache.org/viewvc?view=revision&revision=1381124 LUCENE-4360 : Support running the same test suite multiple times in parallel.
        Hide
        Uwe Schindler added a comment -

        Closed after release.

        Show
        Uwe Schindler added a comment - Closed after release.

          People

          • Assignee:
            Dawid Weiss
            Reporter:
            Dawid Weiss
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development