[CASSANDRA-16507] Node unreachable to each other with MUTATION message dropped

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Invalid
• Component/s: Messaging/Internode
• Platform: All
• Assignee: Unassigned
• Reporter: Yixin Zou
• Created: 09/Mar/21 08:37
• Updated: 09/Mar/21 19:57
• Resolved: 09/Mar/21 19:57

Description

Cluster setup: Ubuntu 18.04-LTS, cassandra-3.11.10. One DC, 3 nodes, 2 replication factor;

Using an injector to filter and drop MUTATION message between one node and the other two (drop MUTATION messages sent from node 1 to node 2, 3 and drop MUTATION messages sent from node 2, 3 to node 1). At the same time, inserted same key, different values on node 1 and node 2. When insertion happened, MUTATION message drops were caught on both node 1 and node2. Then we observed the different values for that key on two nodes and observed all three nodes disconnected from each other. 

 

./nodetool describecluster

/Cluster Information:Cluster Information: 
Name: Test Cluster Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch DynamicEndPointSnitch: enabled Partitioner: org.apache.cassandra.dht.Murmur3Partitioner 
Schema versions: UNREACHABLE: [10.0.1.8, 10.0.1.7, 10.0.1.9]