[CASSANDRA-18766] high speculative retries on v4.1.3

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 4.1.6
• Component/s: Consistency/Coordination
• Bug Category: Degradation - Performance Bug/Regression
• Severity: Critical
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 4.1.3

Source Control Link

https://github.com/apache/cassandra-dtest/commit/f14da069295aa66e6d1dd638a73ce136e759b1bf

Test and Documentation Plan

add test

• Assignee: Brandon Williams
• Reporter: Ivans Novikovs

Authors

Brandon Williams

Reviewers

Stefan Miklosovic

• Created: 16/Aug/23 14:08
• Updated: 16/Aug/24 18:37
• Resolved: 16/Aug/24 18:37

Description

There are up to 10+ times higher speculative retries for reads on 4.1.3 comparing to 4.0.7 and 4.1.2 when using QUORUM and default setting of 99p.

On 4.1.3 after upgrade I see speculative retries for up to 35% of all reads for specific table. Latency for reads is stable around 500 microseconds.

java 1.8.0_382 is used

Issue Links

• is caused by: CASSANDRA-15241 Virtual table to expose current running queries
• is superceded by: CASSANDRA-19534 Unbounded queues in native transport requests lead to node instability