[KUDU-434] Add/replace nodes in a tablet (handle permanent node failures) - ability to perform manual changes

• Type: New Feature
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: M5
• Fix Version/s: n/a
• Component/s: consensus, tserver
• Labels: roadmap-candidate
• Target Version/s: 1.2.0
• Assignee: Mike Percy
• Reporter: Todd Lipcon
• Due: 15/May/15
• Created: 31/Jul/14 16:14
• Updated: 18/Jun/19 13:01
• Resolved: 16/Aug/16 04:51

Description

This tracks work required to handle a few similar use cases:

• node crashes and doesn't come back ever. a new node needs to be recruited to get a tablet's replication back up to 3
  • equivalently, a node loses its disks and thus is essentially a new node
• add a new node to the cluster, and balance some existing data onto that one

Implementation-wise:

• ability to bootstrap a new node into a quorum (copy existing tablet data)
• ability to change-config a quorum to include a new peer
• some administrative commands to explicitly migrate data to perform manual load balancing (automatic load balancing tracked separately)

Issue Links

• blocks: KUDU-673 Automatic Master management of dynamic consensus configuration