Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-8132

Save or stream hints to a safe place in node replacement

Agile BoardAttach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: In Progress
    • Priority: Normal
    • Resolution: Unresolved
    • Fix Version/s: 4.x
    • Component/s: Consistency/Hints
    • Labels:
      None

      Description

      Often, we need to replace a node with a new instance in the cloud environment where we have all nodes are still alive. To be safe without losing data, we usually make sure all hints are gone before we do this operation.

      Replacement means we just want to shutdown C* process on a node and bring up another instance to take over that node's token.

      However, if a node to be replaced has a lot of stored hints, its HintedHandofManager seems very slow to send the hints to other nodes. In our case, we tried to replace a node and had to wait for several days before its stored hints are clear out. As mentioned above, we need all hints on this node to clear out before we can terminate it and replace it by a new instance/machine.

      Since this is not a decommission, I am proposing that we have the same hints-streaming mechanism as in the decommission code. Furthermore, there needs to be a cmd for NodeTool to trigger this.

        Attachments

          Activity

            People

            • Assignee:
              shaunakdas Shaunak Das Assign to me
              Reporter:
              timiblossom Minh Do
              Authors:
              Shaunak Das

              Dates

              • Due:
                Created:
                Updated:

                Issue deployment