[CASSANDRA-15520] split circle ci commands into reusable scripts which can be used outside of circle ci

• Type: Improvement
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Component/s: Build
• Change Category: Code Clarity
• Complexity: Low Hanging Fruit
• Platform: All
• Assignee: Unassigned
• Reporter: David Capwell
• Created: 24/Jan/20 01:03
• Updated: 30/Jan/20 18:03

Description

CircleCI is one of the main tools we use for build and test, but there is also ASF Jenkins and many people run builds in their own companies as well. It would be nice to refactor the existing CircleCI yaml to delegate to a set of scripts which could be reused by other build systems.

I feel that we could do the following directory layout

ci - top level directory containing all scripts
 - <test type : unit, distributed> - directory containing the different build steps
     - <test name : eg. unit, jvm-dtest, long, etc.> - a single build with the required steps to run it
         - split.sh - script which takes in a output file to write to and dumps out all test cases (not partitioned)
         - run_partition.sh - script which takes a partitioned list of tests and executes the build (does not move around artifacts)

This would allow CircleCI and Jenkins to run the same way, but also acts as documentation for how to run some of the tests (jvm-dtest and python dtest upgrade tests take more time to figure out how to run).

CircleCI would also be simpler as it would mostly be the circle ci specific logic (partition tests, move code/test results around, etc.) calling these scripts.