[LUCENE-9472] True concurrent test suite duplication and execution (beasting)

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Workaround
• Lucene Fields: New
• Assignee: Dawid Weiss
• Reporter: Dawid Weiss
• Created: 20/Aug/20 13:05
• Updated: 28/Aug/22 16:06
• Resolved: 20/Aug/20 15:13

Description

I toyed with this a bit because it's hacker's delight... Forked the default Gradle test task and made it emit the same test/ suite name multiple times. This results in the same test being executed multiple times:

# ./gradlew -p lucene\analysis\kuromoji testy --tests TestToStringUtil.testHepburn -Ptests.dups=10

> Task :randomizationInfo
Running tests with randomization seed: tests.seed=88C21B0E8ABB2D46

> Task :lucene:analysis:kuromoji:testy
:lucene:analysis:kuromoji:testy (SUCCESS): 10 test(s)

BUILD SUCCESSFUL in 13s
21 actionable tasks: 6 executed, 15 up-to-date

Upsides:

• Everything configures itself because the custom task is a subtype of Gradle's Test task,
• Tests are run in parallel, on multiple VMs,
• Opens up the possibility of reordering test suites (for load-balancing).

Downsides:

• Requires a custom(ized) clone of Gradle's internal classes. Will require maintenance when upgrading gradle. There is no certainty it'll work.
• Some things visibly don't like duplicated suite names - the progress bar shows the suite once, even though it is executed concurrently in multiple VMs,
• XML reports will drive jenkins and other tooling crazy:

  <?xml version="1.0" encoding="UTF-8"?>
  <testsuite name="org.apache.lucene.analysis.ja.util.TestToStringUtil" tests="10" skipped="0" failures="0" errors="0" timestamp="2020-08-20T12:54:27" hostname="DWEISS-DESKTOP" time="3.177">
    <properties/>
    <testcase name="testHepburn" classname="org.apache.lucene.analysis.ja.util.TestToStringUtil" time="1.04"/>
    <testcase name="testHepburn" classname="org.apache.lucene.analysis.ja.util.TestToStringUtil" time="1.045"/>
    <testcase name="testHepburn" classname="org.apache.lucene.analysis.ja.util.TestToStringUtil" time="1.043"/>
    <testcase name="testHepburn" classname="org.apache.lucene.analysis.ja.util.TestToStringUtil" time="1.038"/>
    <testcase name="testHepburn" classname="org.apache.lucene.analysis.ja.util.TestToStringUtil" time="1.008"/>
    <testcase name="testHepburn" classname="org.apache.lucene.analysis.ja.util.TestToStringUtil" time="1.005"/>
  ....
  

• not sure what happens with sysouts from such concurrent runs...

Issue Links

• is related to: LUCENE-9465 Add functionality to do test re-runs (suite duplication, so-called 'beasting')
• links to: GitHub Pull Request #1768