[CASSANDRA-17321] Investigate large number of test timeouts on specific build runs

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Component/s: Test/dtest/python
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Assignee: Unassigned
• Reporter: Josh McKenzie
• Created: 28/Jan/22 18:47
• Updated: 17/Apr/23 21:16

Description

We have a selection of new test failures on CI run 323 on 4.0 and 923 on trunk that appear to have a variety of teardown errors or timeouts. We should look into if there's any kind of theme, commonality, etc that we can make changes w/respect to to prevent this kind of "dud job" from occurring in the future.

323 4.0: https://ci-cassandra.apache.org/job/Cassandra-4.0/323/testReport/
923 trunk: https://ci-cassandra.apache.org/job/Cassandra-trunk/923/

Failures on 4.0:
https://ci-cassandra.apache.org/job/Cassandra-4.0/323/testReport/dtest-novnode.materialized_views_test/TestMaterializedViewsConsistency/test_multi_partition_consistent_reads_after_write/
https://ci-cassandra.apache.org/job/Cassandra-4.0/323/testReport/dtest-novnode.transient_replication_test/TestTransientReplication/test_cheap_quorums/
https://ci-cassandra.apache.org/job/Cassandra-4.0/323/testReport/dtest-novnode.client_request_metrics_test/TestClientRequestMetrics/test_client_request_metrics/
https://ci-cassandra.apache.org/job/Cassandra-4.0/323/testReport/dtest-novnode.repair_tests.repair_test/TestRepair/test_multiple_ranges_repair/
https://ci-cassandra.apache.org/job/Cassandra-4.0/323/testReport/org.apache.cassandra.net/ProxyHandlerConnectionsTest/testExpireSome/

Notably, trunk build 923 appears to have the same pattern of a slew of untraced timeouts on tests:
https://ci-cassandra.apache.org/job/Cassandra-trunk/923/testReport/org.apache.cassandra.distributed.upgrade/CompactStorageUpgradeTest/compactStorageHiddenColumnTest/
https://ci-cassandra.apache.org/job/Cassandra-trunk/923/testReport/org.apache.cassandra.transport/CQLConnectionTest/handleCorruptionOfLargeMessageFrame_compression_2/
https://ci-cassandra.apache.org/job/Cassandra-trunk/923/testReport/org.apache.cassandra.distributed.test/NetstatsRepairStreamingTest/testWithCompressionEnabled/
https://ci-cassandra.apache.org/job/Cassandra-trunk/923/testReport/org.apache.cassandra.distributed.test/NetstatsBootstrapWithEntireSSTablesCompressionStreamingTest/testWithStreamingEntireSSTablesWithoutCompressionWithoutThrottling_4/
https://ci-cassandra.apache.org/job/Cassandra-trunk/923/testReport/org.apache.cassandra.net/ProxyHandlerConnectionsTest/testExpireSome_compression/
https://ci-cassandra.apache.org/job/Cassandra-trunk/923/testReport/dtest-novnode.transient_replication_test/TestTransientReplication/test_transient_write/
https://ci-cassandra.apache.org/job/Cassandra-trunk/923/testReport/dtest-novnode.read_repair_test/TestSpeculativeReadRepair/test_quorum_requirement_on_speculated_read/