[CASSANDRA-9753] LOCAL_QUORUM reads can block cross-DC if there is a digest mismatch

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Component/s: Legacy/Coordination
• Severity: Normal
• Since Version: 2.0.11
• Assignee: Unassigned
• Reporter: Richard Low
• Created: 07/Jul/15 21:27
• Updated: 09/Sep/21 02:53

Description

When there is a digest mismatch during the initial read, a data read request is sent to all replicas involved in the initial read. This can be more than the initial blockFor if read repair was done and if speculative retry kicked in. E.g. for RF 3 in two DCs, the number of reads could be 4: 2 for LOCAL_QUORUM, 1 for read repair and 1 for speculative read if one replica was slow. If there is then a digest mismatch, Cassandra will issue the data read to all 4 and set blockFor=4. Now the read query is blocked on cross-DC latency. The digest mismatch read blockFor should be capped at RF for the local DC when using CL.LOCAL_*.

You can reproduce this behaviour by creating a keyspace with NetworkTopologyStrategy, RF 3 per DC, dc_local_read_repair=1.0 and ALWAYS for speculative read. If you force a digest mismatch (e.g. by deleting a replicas SSTables and restarting) you can see in tracing that it is blocking for 4 responses.

Issue Links

• is related to: CASSANDRA-6887 LOCAL_ONE read repair only does local repair, in spite of global digest queries