[CASSANDRA-16006] Parallelise Jenkins dtests

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-beta2, 4.0
• Component/s: CI
• Change Category: Performance
• Complexity: Normal
• Platform: All

Source Control Link

https://github.com/apache/cassandra-builds/commit/6fc260ae265e7c8f8c29445264ae143e1693cf14

Test and Documentation Plan

Tested on local jenkins. Tested on ci-cassandra.a.o with dtest-upgrade jobs.

• Assignee: Michael Semb Wever Assign to me
• Reporter: Michael Semb Wever

Authors

Michael Semb Wever

Reviewers

Eduard Tudenhoefner

• Created: 31/Jul/20 18:27
• Updated: 03/Jan/21 20:03
• Resolved: 05/Aug/20 18:53

Description

Currently dtests in Jenkins take ~10 hours.

Using the Jenkins Matrix plugin these jobs can be split into smaller lists of dtests and run in parallel. This is the approach CircleCI takes.

This approach was trialed with the dtest-upgrade jobs (which are not yet part of the branch pipelines, and haven't previously worked at all due to their duration).

In addition to the Matrix plugin, the Priority-Sorter and Matrix Reloaded plugins also needed to be added.

The splits will occupy all executors, and multiple builds will lead to a long build queue. More important builds (artifacts and unit tests) need a way to be prioritised in such saturated situations.

Splits can fail for silly reasons (false-positive), like full /tmp disks, or connectivity issues between the donated agent servers. The Matrix Reloaded plugin makes it easy to rebuilt just those failed splits.