[CASSANDRA-19495] Hints not stored after node goes down for the second time

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 4.1.5, 5.0-rc1, 5.1-alpha1
• Component/s: Consistency/Hints
• Bug Category: Degradation - Performance Bug/Regression
• Severity: Critical
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 4.1.0

Source Control Link

https://github.com/apache/cassandra/commit/5fb562d7efbad7bf9c2297a62991d36da90589e9

Test and Documentation Plan

ci

• Assignee: Stefan Miklosovic
• Reporter: Paul Chandler

Authors

Stefan Miklosovic

Reviewers

Brandon Williams, Stefan Miklosovic

• Created: 26/Mar/24 17:30
• Updated: 20/May/24 11:31
• Resolved: 05/Apr/24 09:53

Description

I have a scenario where a node goes down, hints are recorded on the second node and replayed, as expected. If the first node goes down for a second time and time span between the first time it stopped and the second time it stopped is more than the max_hint_window then the hint is not recorded, no hint file is created, and the mutation never arrives at the node after it comes up again.

I have debugged this and it appears to due to the way hint window is persisted after https://issues.apache.org/jira/browse/CASSANDRA-14309

The code here: https://github.com/apache/cassandra/blame/cassandra-4.1/src/java/org/apache/cassandra/service/StorageProxy.java#L2402 uses the time stored in the HintsBuffer.earliestHintByHost map.  This map is based on the UUID of the host, but this does not seem to be cleared when the node is back up, and I think this is what is causing the problem.

 

This is in cassandra 4.1.5

Issue Links

• links to: GitHub Pull Request #3206
• links to: GitHub Pull Request #3211
• links to: GitHub Pull Request #3213
• links to: GitHub Pull Request #3278