[CASSANDRA-19365] invalid EstimatedHistogramReservoirSnapshot::getValue values due to race condition in DecayingEstimatedHistogramReservoir

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.1.8, 5.0.2, 5.x
• Component/s: Observability/Metrics
• Bug Category: Correctness
• Severity: Normal
• Complexity: Normal
• Discovered By: Adhoc Test
• Platform: All
• Since Version: 4.0

Source Control Link

https://github.com/apache/cassandra/commit/ad4ad5ef5aa9a432d7126140fc9f4fedcd20411c

Test and Documentation Plan

Documentation is not required. Run tests to verify the patch.

• Assignee: Maxim Muzafarov Assign to me
• Reporter: Jakub Zytka

Authors

Jakub Zytka, Maxim Muzafarov

Reviewers

Caleb Rackliffe

• Created: 05/Feb/24 15:19
• Updated: 02/Oct/24 15:04
• Resolved: 02/Oct/24 13:46

Description

`DecayingEstimatedHistogramReservoir` has a race condition between `update` and `rescaleIfNeeded`.
A sample which ends up (`update`) in an already scaled decayingBucket (`rescaleIfNeeded`) may still use a non-scaled weight because `decayLandmark` has not been updated yet at the moment of `update`.
 
The observed consequence was flooding of the cluster with speculative retries (we happened to hit low-percentile buckets with overweight samples, which drove p99 below true p50 for a long time).

Please note that despite the manifestation being similar to CASSANDRA-19330, these are two distinct bugs in their own right.

This bug affects versions 4.0+
On 3.11 there's locking in DEHR. I did not check earlier versions.

Issue Links

• links to: GitHub Pull Request #3543