[CASSANDRA-17930] Ensure CircleCI and ASF Jenkins CI are aligned

• Type: Task
• Status: In Progress
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 3.0.x, 3.11.x, 4.0.x, 4.1.x, 5.x
• Component/s: CI
• Epic Link: circleci_improvements
• Change Category: Quality Assurance
• Complexity: Normal
• Platform: All
• Assignee: Unassigned
• Reporter: Ekaterina Dimitrova
• Created: 27/Sep/22 16:57
• Updated: 08/Jun/23 12:38

Description

As discussed in this thread the Cassandra community wants to see CircleCI and ASF CI being aligned - running the same tests, configurations, all tests.

Exceptions: packaging.

A few examples of discrepancies we already noticed:

• utests_system_keyspace_directory run only in CircleCI - CASSANDRA-17145
• dtest-large run only in Jenkins
• simulator tests run only in CircleCI
• In a quick skim I think I didn't see these runs too in CircleCI - dtest-offheap, dtest-large, dtest-large-novnode
• packaging is also only tested in CircleCI as far as I recall but that is deterministic and we will rely on Jenkins about that. 

And these are only a few examples on top of my mind. I am sure we will find more. We also need to verify the way we call those tests is correct and matches in both CIs. (I was looking to solve similar discrepancy in CASSANDRA-17912)

Some info on our tests suites here - https://cassandra.apache.org/_/development/testing.html,

 cassandra-builds repo where our test images reside and the Jenkins build scripts, which I already referred to. 

CircleCI info can be found in the readme which resides in the in-tree folder dedicated to configuration and scripts for Circle CI - https://github.com/apache/cassandra/tree/trunk/.circleci

EDIT: More findings:

•  burn tests missing in CircleIC
• cqlshlib in 3.11.
• 4.0+ we also need in-jvm JDK8 with 11, cqlshlib JDK8 with 11.

EDIT: Java distributed tests running with vnodes should be added to Jenkins

 

Issue Links

• is a parent of: CASSANDRA-17671 Run utests_system_keyspace_directory with Java 11 in CircleCI
• is a parent of: CASSANDRA-17987 CircleCI: Add jobs for running specialized unit tests with Java 11
• is a parent of: CASSANDRA-17989 Jenkins: Add unit tests for trie-based memtables
• is related to: CASSANDRA-17939 CircleCI: Automatically detect and repeat new or modified JUnit tests
• is related to: CASSANDRA-18008 Jenkins: Add java distributed tests running with vnodes
• relates to: CASSANDRA-17912 Fix CircleCI config for running Python upgrade tests on 3.0 and 3.11
• relates to: CASSANDRA-17145 utests_system_keyspace_directory is not run in Jenkins
• relates to: CASSANDRA-18101 Jenkins: cqlsh tests should be run with all Python versions they are run in CircleCI
• relates to: CASSANDRA-18001 Add missing tests suites to CircleCI
• relates to: CASSANDRA-18003 Add simulator tests to Jenkins

Sub-Tasks

• Enable dtest-offheap in CircleCI: Resolved