Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-4360

Support running the same test suite multiple times in parallel

Details

    • New Feature
    • Status: Closed
    • Trivial
    • Resolution: Fixed
    • None
    • 4.0, 6.0
    • general/test
    • None
    • New

    Description

      The current "test execution multiplier" or:

      -Dtests.iters=N
      

      generates multiple tests (method executions) under a test class (suite). All these tests, however, are bound to a single class so they must run sequentially and on a single JVM (because of how JUnit works – nesting of rules, class hooks, etc.).

      Mark pointed out that if somebody has a multi-core CPU then it'd be nice to be able to run a single suite in parallel, possibly in combination with tests.iters (so that a single test method is executed X times on Y parallel JVMs).

      This is surprisingly easy with the randomized runner because it currently accepts "duplicate" suite names and will load-balance them in a normal way. So, if one has Y cores (JVMs) then providing a suite name X times will result in X executions, balanced across Y JVMs.

      The only problem is how to "multiply" suite names. This can be done in a number of ways, starting from a custom resource collection wrapper and ending at a built-in code in the runner itself. I think the custom collection wrapper approach would be interesting, I'll explore this direction.

      Attachments

        1. quickhack.patch
          1 kB
          Dawid Weiss

        Activity

          dweiss Dawid Weiss added a comment -

          This is a quick hack that multiplies execution of all tests by N-fold by just repeating the same fileset a few times.

          A proof of concept, but working.

          dweiss Dawid Weiss added a comment - This is a quick hack that multiplies execution of all tests by N-fold by just repeating the same fileset a few times. A proof of concept, but working.
          dweiss Dawid Weiss added a comment -

          Any ideas what to call this parameter? It's not the same as tests.iters because it multiplies entire suites. I came up with:

          -Dtests.multiply
          -Dtests.duplicate
          
          or the short equivalents:
          
          -Dtests.mply
          -Dtests.dups
          
          dweiss Dawid Weiss added a comment - Any ideas what to call this parameter? It's not the same as tests.iters because it multiplies entire suites. I came up with: -Dtests.multiply -Dtests.duplicate or the short equivalents: -Dtests.mply -Dtests.dups
          dweiss Dawid Weiss added a comment -

          Example of use:

          ant test-core -Dtests.dups=3 -Dtestcase=TestCharFilter
          
          <JUnit4> says kaixo! Master seed: 4CE0022F3CDFF5D7
          Your default console's encoding may not display certain unicode glyphs: windows-1252
          Executing 3 suites with 3 JVMs.
          
          Suite: org.apache.lucene.analysis.TestCharFilter
          OK      0.05s J1 | TestCharFilter.testCharFilter1
          OK      0.01s J1 | TestCharFilter.testCharFilter11
          OK      0.00s J1 | TestCharFilter.testCharFilter2
          OK      0.00s J1 | TestCharFilter.testCharFilter12
          Completed on J1 in 0.35s, 4 tests
          
          Suite: org.apache.lucene.analysis.TestCharFilter
          OK      0.04s J2 | TestCharFilter.testCharFilter1
          OK      0.01s J2 | TestCharFilter.testCharFilter11
          OK      0.00s J2 | TestCharFilter.testCharFilter2
          OK      0.01s J2 | TestCharFilter.testCharFilter12
          Completed on J2 in 0.35s, 4 tests
          
          Suite: org.apache.lucene.analysis.TestCharFilter
          OK      0.04s J0 | TestCharFilter.testCharFilter1
          OK      0.02s J0 | TestCharFilter.testCharFilter11
          OK      0.00s J0 | TestCharFilter.testCharFilter2
          OK      0.00s J0 | TestCharFilter.testCharFilter12
          Completed on J0 in 0.36s, 4 tests
          
          JVM J0:     0.67 ..     1.56 =     0.89s
          JVM J1:     0.67 ..     1.55 =     0.88s
          JVM J2:     0.67 ..     1.57 =     0.90s
          Execution time total: 1.58 sec.
          Tests summary: 3 suites, 12 tests
          

          In combination with tests.iters:

          ant test-core -Dtests.dups=2 -Dtests.iters=2 -Dtestcase=TestCharFilter
          
          <JUnit4> says Ă­Hola! Master seed: 592442048D98540B
          Your default console's encoding may not display certain unicode glyphs: windows-1252
          Executing 2 suites with 2 JVMs.
          
          Suite: org.apache.lucene.analysis.TestCharFilter
          OK      0.04s J1 | TestCharFilter.testCharFilter12 {#0 seed=[592442048D98540B:5E8780464DC70845]}
          OK      0.00s J1 | TestCharFilter.testCharFilter12 {#1 seed=[592442048D98540B:EAD13CBA7905C369]}
          OK      0.00s J1 | TestCharFilter.testCharFilter11 {#0 seed=[592442048D98540B:4AC9FEB2457325F4]}
          OK      0.00s J1 | TestCharFilter.testCharFilter11 {#1 seed=[592442048D98540B:FE9F424E71B1EED8]}
          OK      0.00s J1 | TestCharFilter.testCharFilter1 {#0 seed=[592442048D98540B:643276B2D133E783]}
          OK      0.00s J1 | TestCharFilter.testCharFilter1 {#1 seed=[592442048D98540B:D064CA4EE5F12CAF]}
          OK      0.00s J1 | TestCharFilter.testCharFilter2 {#0 seed=[592442048D98540B:D339FCA65F87D49C]}
          OK      0.01s J1 | TestCharFilter.testCharFilter2 {#1 seed=[592442048D98540B:676F405A6B451FB0]}
          Completed on J1 in 0.34s, 8 tests
          
          Suite: org.apache.lucene.analysis.TestCharFilter
          OK      0.05s J0 | TestCharFilter.testCharFilter12 {#0 seed=[592442048D98540B:5E8780464DC70845]}
          OK      0.00s J0 | TestCharFilter.testCharFilter12 {#1 seed=[592442048D98540B:EAD13CBA7905C369]}
          OK      0.00s J0 | TestCharFilter.testCharFilter11 {#0 seed=[592442048D98540B:4AC9FEB2457325F4]}
          OK      0.00s J0 | TestCharFilter.testCharFilter11 {#1 seed=[592442048D98540B:FE9F424E71B1EED8]}
          OK      0.00s J0 | TestCharFilter.testCharFilter1 {#0 seed=[592442048D98540B:643276B2D133E783]}
          OK      0.00s J0 | TestCharFilter.testCharFilter1 {#1 seed=[592442048D98540B:D064CA4EE5F12CAF]}
          OK      0.00s J0 | TestCharFilter.testCharFilter2 {#0 seed=[592442048D98540B:D339FCA65F87D49C]}
          OK      0.00s J0 | TestCharFilter.testCharFilter2 {#1 seed=[592442048D98540B:676F405A6B451FB0]}
          Completed on J0 in 0.35s, 8 tests
          
          JVM J0:     0.66 ..     1.49 =     0.82s
          JVM J1:     0.66 ..     1.28 =     0.62s
          Execution time total: 1.49 sec.
          Tests summary: 2 suites, 16 tests
          

          Note each JVM will start from the same master seed. Each test will work in its own working directory and JVM though so it can be useful for performing stress testing of a single suite.

          dweiss Dawid Weiss added a comment - Example of use: ant test-core -Dtests.dups=3 -Dtestcase=TestCharFilter <JUnit4> says kaixo! Master seed: 4CE0022F3CDFF5D7 Your default console's encoding may not display certain unicode glyphs: windows-1252 Executing 3 suites with 3 JVMs. Suite: org.apache.lucene.analysis.TestCharFilter OK 0.05s J1 | TestCharFilter.testCharFilter1 OK 0.01s J1 | TestCharFilter.testCharFilter11 OK 0.00s J1 | TestCharFilter.testCharFilter2 OK 0.00s J1 | TestCharFilter.testCharFilter12 Completed on J1 in 0.35s, 4 tests Suite: org.apache.lucene.analysis.TestCharFilter OK 0.04s J2 | TestCharFilter.testCharFilter1 OK 0.01s J2 | TestCharFilter.testCharFilter11 OK 0.00s J2 | TestCharFilter.testCharFilter2 OK 0.01s J2 | TestCharFilter.testCharFilter12 Completed on J2 in 0.35s, 4 tests Suite: org.apache.lucene.analysis.TestCharFilter OK 0.04s J0 | TestCharFilter.testCharFilter1 OK 0.02s J0 | TestCharFilter.testCharFilter11 OK 0.00s J0 | TestCharFilter.testCharFilter2 OK 0.00s J0 | TestCharFilter.testCharFilter12 Completed on J0 in 0.36s, 4 tests JVM J0: 0.67 .. 1.56 = 0.89s JVM J1: 0.67 .. 1.55 = 0.88s JVM J2: 0.67 .. 1.57 = 0.90s Execution time total: 1.58 sec. Tests summary: 3 suites, 12 tests In combination with tests.iters: ant test-core -Dtests.dups=2 -Dtests.iters=2 -Dtestcase=TestCharFilter <JUnit4> says Ă­Hola! Master seed: 592442048D98540B Your default console's encoding may not display certain unicode glyphs: windows-1252 Executing 2 suites with 2 JVMs. Suite: org.apache.lucene.analysis.TestCharFilter OK 0.04s J1 | TestCharFilter.testCharFilter12 {#0 seed=[592442048D98540B:5E8780464DC70845]} OK 0.00s J1 | TestCharFilter.testCharFilter12 {#1 seed=[592442048D98540B:EAD13CBA7905C369]} OK 0.00s J1 | TestCharFilter.testCharFilter11 {#0 seed=[592442048D98540B:4AC9FEB2457325F4]} OK 0.00s J1 | TestCharFilter.testCharFilter11 {#1 seed=[592442048D98540B:FE9F424E71B1EED8]} OK 0.00s J1 | TestCharFilter.testCharFilter1 {#0 seed=[592442048D98540B:643276B2D133E783]} OK 0.00s J1 | TestCharFilter.testCharFilter1 {#1 seed=[592442048D98540B:D064CA4EE5F12CAF]} OK 0.00s J1 | TestCharFilter.testCharFilter2 {#0 seed=[592442048D98540B:D339FCA65F87D49C]} OK 0.01s J1 | TestCharFilter.testCharFilter2 {#1 seed=[592442048D98540B:676F405A6B451FB0]} Completed on J1 in 0.34s, 8 tests Suite: org.apache.lucene.analysis.TestCharFilter OK 0.05s J0 | TestCharFilter.testCharFilter12 {#0 seed=[592442048D98540B:5E8780464DC70845]} OK 0.00s J0 | TestCharFilter.testCharFilter12 {#1 seed=[592442048D98540B:EAD13CBA7905C369]} OK 0.00s J0 | TestCharFilter.testCharFilter11 {#0 seed=[592442048D98540B:4AC9FEB2457325F4]} OK 0.00s J0 | TestCharFilter.testCharFilter11 {#1 seed=[592442048D98540B:FE9F424E71B1EED8]} OK 0.00s J0 | TestCharFilter.testCharFilter1 {#0 seed=[592442048D98540B:643276B2D133E783]} OK 0.00s J0 | TestCharFilter.testCharFilter1 {#1 seed=[592442048D98540B:D064CA4EE5F12CAF]} OK 0.00s J0 | TestCharFilter.testCharFilter2 {#0 seed=[592442048D98540B:D339FCA65F87D49C]} OK 0.00s J0 | TestCharFilter.testCharFilter2 {#1 seed=[592442048D98540B:676F405A6B451FB0]} Completed on J0 in 0.35s, 8 tests JVM J0: 0.66 .. 1.49 = 0.82s JVM J1: 0.66 .. 1.28 = 0.62s Execution time total: 1.49 sec. Tests summary: 2 suites, 16 tests Note each JVM will start from the same master seed. Each test will work in its own working directory and JVM though so it can be useful for performing stress testing of a single suite.
          siren Sami Siren added a comment -

          Cool! This is a really nice addition!

          siren Sami Siren added a comment - Cool! This is a really nice addition!
          dweiss Dawid Weiss added a comment -

          Thanks! It was Mark's request, actually. I had all the infrastructure ready, I just needed that duplicating resource collection and this was trivial.

          dweiss Dawid Weiss added a comment - Thanks! It was Mark's request, actually. I had all the infrastructure ready, I just needed that duplicating resource collection and this was trivial.
          commit-tag-bot Commit Tag Bot added a comment -

          [branch_4x commit] Dawid Weiss
          http://svn.apache.org/viewvc?view=revision&revision=1381124

          LUCENE-4360: Support running the same test suite multiple times in parallel.

          commit-tag-bot Commit Tag Bot added a comment - [branch_4x commit] Dawid Weiss http://svn.apache.org/viewvc?view=revision&revision=1381124 LUCENE-4360 : Support running the same test suite multiple times in parallel.
          uschindler Uwe Schindler added a comment -

          Closed after release.

          uschindler Uwe Schindler added a comment - Closed after release.
          tomoko Tomoko Uchida added a comment -

          This issue was moved to GitHub issue: #5426.

          tomoko Tomoko Uchida added a comment - This issue was moved to GitHub issue: #5426 .

          People

            dweiss Dawid Weiss
            dweiss Dawid Weiss
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: