Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-9753

LOCAL_QUORUM reads can block cross-DC if there is a digest mismatch

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Normal
    • Resolution: Unresolved
    • None
    • Legacy/Coordination
    • None
    • Normal

    Description

      When there is a digest mismatch during the initial read, a data read request is sent to all replicas involved in the initial read. This can be more than the initial blockFor if read repair was done and if speculative retry kicked in. E.g. for RF 3 in two DCs, the number of reads could be 4: 2 for LOCAL_QUORUM, 1 for read repair and 1 for speculative read if one replica was slow. If there is then a digest mismatch, Cassandra will issue the data read to all 4 and set blockFor=4. Now the read query is blocked on cross-DC latency. The digest mismatch read blockFor should be capped at RF for the local DC when using CL.LOCAL_*.

      You can reproduce this behaviour by creating a keyspace with NetworkTopologyStrategy, RF 3 per DC, dc_local_read_repair=1.0 and ALWAYS for speculative read. If you force a digest mismatch (e.g. by deleting a replicas SSTables and restarting) you can see in tracing that it is blocking for 4 responses.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            rlow Richard Low

            Dates

              Created:
              Updated:

              Slack

                Issue deployment