[CASSANDRA-16604] Parallelise docker container runs for tests in ci-cassandra.a.o

• Type: Task
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.2.20, 3.0.25, 3.11.11, 4.0-rc2, 4.0, 4.1-alpha1, 4.1
• Component/s: CI, Test/unit
• Change Category: Performance
• Complexity: Normal
• Platform: All

Source Control Link

https://github.com/apache/cassandra-builds/commit/4ca24701e105226ca35da1adc7ff70b56a36fce2 https://github.com/apache/cassandra-builds/commit/1b453f1008696eaac832071b486ae35104f5b255 https://github.com/apache/cassandra-builds/commit/a351b781c9b7fa054d6892

Test and Documentation Plan

CI

• Assignee: Michael Semb Wever
• Reporter: Michael Semb Wever

Authors

Michael Semb Wever

Reviewers

Tomasz Lasica

• Created: 15/Apr/21 06:41
• Updated: 27/May/22 19:24
• Resolved: 06/May/21 10:00

Description

This was raised on the dev ML, where the consensus was to remove it: https://lists.apache.org/thread.html/r1ca3c72b90fa6c57c1cb7dcd02a44221dcca991fe7392abd8c29fe95%40%3Cdev.cassandra.apache.org%3E

The idea is to then replace ant test parallelism with docker container parallelism.

PoC patch:
https://github.com/apache/cassandra-builds/compare/trunk...thelastpickle:mck/16587-2/trunk

This is just a quick PoC, aimed at the ci-cassandra agents that have
4 cores and 16gb ram available to each executor, but I imagine instead
something that spawns a number of containers based on system
resources, like we currently do with get-cores and get-mem.

Also worth noting the overhead here, compared with the ant parallelism approach, docker
builds everything in each container from scratch, but this too can be
improved easily enough.

Cleaning up any remnant `-Dtest.runners=` options is also part of this ticket.

Issue Links

• is related to: CASSANDRA-16595 Remove test parallelism from ant build.xml in all branches