Details
-
New Feature
-
Status: In Progress
-
Normal
-
Resolution: Unresolved
-
None
Description
Often, we need to replace a node with a new instance in the cloud environment where we have all nodes are still alive. To be safe without losing data, we usually make sure all hints are gone before we do this operation.
Replacement means we just want to shutdown C* process on a node and bring up another instance to take over that node's token.
However, if a node to be replaced has a lot of stored hints, its HintedHandofManager seems very slow to send the hints to other nodes. In our case, we tried to replace a node and had to wait for several days before its stored hints are clear out. As mentioned above, we need all hints on this node to clear out before we can terminate it and replace it by a new instance/machine.
Since this is not a decommission, I am proposing that we have the same hints-streaming mechanism as in the decommission code. Furthermore, there needs to be a cmd for NodeTool to trigger this.