Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-9753

LOCAL_QUORUM reads can block cross-DC if there is a digest mismatch

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Normal
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: Legacy/Coordination
    • Labels:
      None
    • Severity:
      Normal
    • Since Version:

      Description

      When there is a digest mismatch during the initial read, a data read request is sent to all replicas involved in the initial read. This can be more than the initial blockFor if read repair was done and if speculative retry kicked in. E.g. for RF 3 in two DCs, the number of reads could be 4: 2 for LOCAL_QUORUM, 1 for read repair and 1 for speculative read if one replica was slow. If there is then a digest mismatch, Cassandra will issue the data read to all 4 and set blockFor=4. Now the read query is blocked on cross-DC latency. The digest mismatch read blockFor should be capped at RF for the local DC when using CL.LOCAL_*.

      You can reproduce this behaviour by creating a keyspace with NetworkTopologyStrategy, RF 3 per DC, dc_local_read_repair=1.0 and ALWAYS for speculative read. If you force a digest mismatch (e.g. by deleting a replicas SSTables and restarting) you can see in tracing that it is blocking for 4 responses.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                rlow Richard Low
              • Votes:
                3 Vote for this issue
                Watchers:
                23 Start watching this issue

                Dates

                • Created:
                  Updated: