[CASSANDRA-9641] Occasional timeouts with blockFor=all for LOCAL_QUORUM query

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 2.1.x, 2.2.x, 3.0.x
• Component/s: Legacy/Coordination
• Severity: Normal
• Since Version: 2.0.11
• Assignee: Unassigned
• Reporter: Richard Low
• Created: 23/Jun/15 19:01
• Updated: 16/Apr/19 09:31

Description

We have a keyspace using NetworkTopologyStrategy with options DC1:3, DC2:3. Our tables have

read_repair_chance = 0.0
dclocal_read_repair_chance = 0.1
speculative_retry = ’99.0PERCENTILE'

and all reads are at LOCAL_QUORUM. On 2.0.11, we occasionally see this timeout:

Cassandra timeout during read query at consistency ALL (6 responses were required but only 5 replica responded)

(sometimes only 4 respond). The ALL is probably due to CASSANDRA-7947 if this occurs during a digest mismatch, but what is interesting is it is expecting 6 responses i.e. blockFor is set to all replicas. I can’t see how this should happen. From the code it should never set blockFor to more than 4 (although 4 is still wrong - I'll make a separate JIRA for that).