[CASSANDRA-20118] Hints ignored during Upgrade from C*4 to C*5

• Type: Bug
• Status: Patch Available
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 5.0.x
• Component/s: Consistency/Hints
• Bug Category: Degradation
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All

Test and Documentation Plan

run CI

• Assignee: Brandon Williams
• Reporter: Paul Chandler

Authors

Brandon Williams

• Created: 27/Nov/24 15:23
• Updated: 17/Dec/24 09:39

Description

I have discovered that some hints were not being processed after nodes come back up when a cluster in in a mixed mode with some cassandra 4 nodes and some cassdandra 5 nodes ( these with a storage compatibility mode CASSANDRA_4 )

 

When in this mode there is a schema mismatch after the first node has been upgraded, which continues until the last node has been upgraded.

It seems that the hints are blocked from being sent if there is a schema mismatch between the 2 nodes, that can be seen at this line.

cassandra/src/java/org/apache/cassandra/hints/HintsDispatchTrigger.java at cassandra-5.0 · apache/cassandra

I have tested removing this line, and that then does allow the hint to be transferred normally. However I am not sure of the implications for doing that if the hint is for part of the schema where the actual mismatch occurs.

 

This creates the problem when a node is being upgraded and is currently down, hint files will be created for it on the new cassandra 5 nodes and the old cassandra 4 nodes, but the hint files on the old cassandra 4 nodes will not be processed, due to the schema mismatch. Leading to potential data loss. 

Issue Links

• links to: GitHub Pull Request #3749
• links to: Github repo