Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Not A Problem
-
None
-
None
-
None
-
Cassandra 3.7
Single DC
7 Nodes
RF 3
NetworkTopologyStrategy
OS: Ubuntu
-
Normal
Description
We experienced strange behaviour of our C* cluster last week.
In the logs we've seen multiple requests in the logs where we the values read from Cassandra actually have been 'null'.
When running the query on a node, we sometimes received the result and sometimes not.
When setting the CL to LOCAL_QUORUM everything was fine. So far nothing unusual, probably the dataset wasn't replicated to one of the nodes.
When turning on the tracing and running the query there was following intersting line:
Initiating read-repair [SharedPool-Worker-2] | 2016-11-18 10:17:47.528000 | $PUBLIC_IP | 126 | 127.0.0.1
Digest mismatch: org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(-5887526567589486157, 3130333031303338383436303937) (db1e86d507513ff12ba95f0eff984b60 vs d41d8cd98f00b204e9800998ecf8427e) [ReadRepairStage:1]
This is probably related to CASSANDRA-12090?
The interesting part is, after that we've run a 'nodetool repair -pr', after that the behaviour was still the same and the data randomly not available, depending on which node was hit.
Only after running a 'nodetool repair -pr -full' the issue was gone.
Did we miss something here? The point that's bothering me is that the dataset was not replicated.
Worth to note is probably that some weeks ago we've hit the bug CASSANDRA-12694 and fixed it by scrubbing some tables.
Any hints/help are appreciated.