[CASSANDRA-16434] Data mismatch across the nodes in the cluster

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Invalid
• Component/s: Consistency/Repair
• Platform: All
• Assignee: Unassigned
• Reporter: Muhammad
• Created: 09/Feb/21 15:23
• Updated: 09/Feb/21 15:30
• Resolved: 09/Feb/21 15:30

Description

Hi,

 

Recently I have came across one strange issue. On an existing cluster with 12 nodes, there were new nodes launched replacing the older ones. It was ensured that only one node was being added or removed at a time until it comes to healthy state. Once the process was completed, we found some data consistency issues in the cluster. Querying a table from some nodes gave back null results which wasn't expected. Is this expected behaviour? We are running Cassandra 3.11.3 version.

Though this problem was solved by running repairs, but it doesn't sound right. There wasn't any strange thing observed from the Cassandra logs (just a few timeouts spread over time which is quite normal when a cluster is running).