Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-9753

LOCAL_QUORUM reads can block cross-DC if there is a digest mismatch

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Normal
    • Resolution: Unresolved
    • None
    • Legacy/Coordination
    • None
    • Normal

    Description

      When there is a digest mismatch during the initial read, a data read request is sent to all replicas involved in the initial read. This can be more than the initial blockFor if read repair was done and if speculative retry kicked in. E.g. for RF 3 in two DCs, the number of reads could be 4: 2 for LOCAL_QUORUM, 1 for read repair and 1 for speculative read if one replica was slow. If there is then a digest mismatch, Cassandra will issue the data read to all 4 and set blockFor=4. Now the read query is blocked on cross-DC latency. The digest mismatch read blockFor should be capped at RF for the local DC when using CL.LOCAL_*.

      You can reproduce this behaviour by creating a keyspace with NetworkTopologyStrategy, RF 3 per DC, dc_local_read_repair=1.0 and ALWAYS for speculative read. If you force a digest mismatch (e.g. by deleting a replicas SSTables and restarting) you can see in tracing that it is blocking for 4 responses.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              rlow Richard Low
              Votes:
              3 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

                Created:
                Updated: