[CASSANDRA-16877] High priority internode messages which exceed the large message threshold are dropped

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0.1
• Component/s: Cluster/Gossip
• Bug Category: Availability - Process Crash
• Severity: Critical
• Complexity: Normal
• Discovered By: Adhoc Test
• Platform: All
• Since Version: 4.0-alpha1

Source Control Link

https://github.com/apache/cassandra/commit/b8242730918c2e8edec83aeafeeae8255378125d

Test and Documentation Plan

New unit test in patch

• Assignee: Sam Tunnicliffe
• Reporter: Sam Tunnicliffe

Authors

Sam Tunnicliffe

Reviewers

Caleb Rackliffe

• Created: 23/Aug/21 17:05
• Updated: 25/Aug/21 16:47
• Resolved: 25/Aug/21 16:47

Description

Currently, there is an assumption that internode messages whose verb has priority P0 will always fit within a single messaging frame. While this is usually the case, on occasion it is possible that this assumption does not hold. One example is gossip messages during the startup shadow round, where in very large clusters the digest ack can contain all states for every peer. In this scenario, the respondent fails to send the ack which may lead to the shadow round and, ultimately, the startup failing.
 
We could tweak the shadow round acks to minimise the message size, but a more robust solution would be to permit high priority messages to be sent on the large messages connection when necessary.