[CASSANDRA-19813] timeout on upgrade_tests/upgrade_through_versions_test.py::TestProtoV3Upgrade_AllVersions_RandomPartitioner_EndsAt_Trunk_HEAD::test_parallel_upgrade

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 5.0.1, 5.1
• Component/s: CI
• Bug Category: Degradation - Slow Use Case
• Severity: Critical
• Complexity: Challenging
• Discovered By: DTest
• Platform: All
• Since Version: 5.0-rc1

Source Control Link

https://github.com/apache/cassandra/commit/f3e7413159d737a41a82750b9c93b44274575337

Test and Documentation Plan

ci

• Assignee: Michael Semb Wever
• Reporter: Michael Semb Wever

Authors

Michael Semb Wever

Reviewers

Brandon Williams

• Created: 02/Aug/24 10:07
• Updated: 07/Aug/24 09:34
• Resolved: 07/Aug/24 09:33

Description

The upgrade_tests/upgrade_through_versions_test.py::TestProtoV3Upgrade_AllVersions_RandomPartitioner_EndsAt_Trunk_HEAD::test_parallel_upgrade test is taking longer than usual.

CI in 5.0 is timing out the dtest-upgrade-large 59/64 split always now.

Our 5.0-rc testing results are tainted because these time outs always abort the 5.0 pipeline runs.

It looks like that split is taking 16x times as long now…

This appears to be caused from either/both

• https://github.com/apache/cassandra/commit/08e1fecf36507397cf3122d77f84aa23150da588
• https://github.com/apache/cassandra-dtest/commit/2b17c1293056068bb3e94c332d6fb99df6a0b0fa

Example of good run.
test_rolling_upgrade.257.good.log (ci-cassandra.a.o)

Examples of bad runs.
test_rolling_upgrade.122.log, test_rolling_upgrade.123-2.log, test_rolling_upgrade.123-1.log, test_rolling_upgrade.134.log (internal ci)
test_rolling_upgrade.261-1.log , test_rolling_upgrade.261-2.log (ci-cassandra.a.o)