Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-17324

Allow node to reject internode messages that create work for the MUTATION stage

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Normal
    • Resolution: Unresolved
    • 5.x
    • Messaging/Internode
    • None
    • Operability
    • Normal
    • All
    • None

    Description

      When a node is struggling under the weight of a compaction backlog and becomes a cause of increased read latency for clients, we have two safety valves:

      1.) Disabling the native protocol server, which stops the node from coordinating reads and writes.
      2.) Jacking up the severity on the node, which tells the dynamic snitch to avoid the node for reads from other coordinators.

      These are useful, but we don’t appear to have any mechanism that would allow us to temporarily reject internode hint, batch, and mutation messages that could further delay resolution of the compaction backlog. There is a parameter in cassandra.yaml called hinted_handoff_throttle (formerly hinted_handoff_throttle_in_kb) that allows us to control the rate at which we read hints before they are delivered, but how fast that should happen and whether it should happen at all are two different questions.

      The proposal here is to add this rejection mechanism and publish it via JMX, along with any metrics and logging that would be necessary to make its effects visible. (Ex. Hint delivery already has metrics around success, failure, and timeouts, which would be helpful around this.) The error-handling pathways for hints, writes, and batches should already be capable of handling one more type of error (i.e. “that replica is overloaded”), but some non-spammy logging around that probably wouldn’t hurt.

      In implementation space, one idea that would minimize the amount of surgery we need to do is making the decision around whether to send back a failure message directly in InboundSink. This would avoid having to duplicate the logic in multiple downstream handlers.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            maedhroz Caleb Rackliffe Assign to me
            maedhroz Caleb Rackliffe
            Caleb Rackliffe

            Dates

              Created:
              Updated:

              Slack

                Issue deployment