[KUDU-2912] Document zero-downtime workflow for 'forgetting' dead tservers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Information Provided
• Affects Version/s: 1.11.0
• Fix Version/s: n/a
• Component/s: documentation
• Assignee: Unassigned
• Reporter: Adar Dembo
• Created: 01/Aug/19 03:45
• Updated: 13/Sep/23 01:57
• Resolved: 13/Sep/23 01:56

Description

This is a fairly useful workflow when the goal is to rebalance the cluster. All it takes is one dead tserver (supposing it's decommissioned and long gone) for rebalancing to refuse to run. As of 1.10.0 there's a CLI parameter that instructs the rebalancer to ignore certain tservers, but it's annoying to put together a UUID list when multiple tservers are dead.

Anyway, the zero-downtime workflow is:

1. Restart all of the masters in the cluster one by one.
2. After each restart, wait for the restarted master to load its tablet and join consensus (ksck should be able to indicate when this was achieved).