Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-14459

DynamicEndpointSnitch should never prefer latent nodes

Agile BoardAttach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete CommentsNeeds ReviewerStart Review
    XMLWordPrintableJSON

    Details

      Description

      The DynamicEndpointSnitch has two unfortunate behaviors that allow it to provide latent hosts as replicas:

      1. Loses all latency information when Cassandra restarts
      2. Clears latency information entirely every ten minutes (by default), allowing global queries to be routed to other datacenters (and local queries cross racks/azs)

      This means that the first few queries after restart/reset could be quite slow compared to average latencies. I propose we solve this by resetting to the minimum observed latency instead of completely clearing the samples and extending the isLatencyForSnitch idea to a three state variable instead of two, in particular YES, NO, MAYBE. This extension allows EchoMessages and PingMessages to send MAYBE indicating that the DS should use those measurements if it only has one or fewer samples for a host. This fixes both problems because on process restart we send out PingMessages / EchoMessages as part of startup, and we would reset to effectively the RTT of the hosts (also at that point normal gossip EchoMessages have an opportunity to add an additional latency measurement).

      This strategy also nicely deals with the "a host got slow but now it's fine" problem that the DS resets were (afaik) designed to stop because the EchoMessage ping latency will count only after the reset for that host. Ping latency is a more reasonable lower bound on host latency (as opposed to status quo of zero).

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              jolynch Joey Lynch Assign to me
              Reporter:
              jolynch Joey Lynch
              Authors:
              Joey Lynch
              Reviewers:
              Ariel Weisberg, Blake Eggleston

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 25.5h
                25.5h

                  Issue deployment