[KUDU-434] Add/replace nodes in a tablet (handle permanent node failures) - ability to perform manual changes

• Type: New Feature
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: M5
• Fix Version/s: n/a
• Component/s: consensus, tserver
• Labels: roadmap-candidate
• Target Version/s: 1.2.0
• Assignee: Mike Percy
• Reporter: Todd Lipcon
• Due: 15/May/15
• Created: 31/Jul/14 16:14
• Updated: 18/Jun/19 13:01
• Resolved: 16/Aug/16 04:51

Description

This tracks work required to handle a few similar use cases:

• node crashes and doesn't come back ever. a new node needs to be recruited to get a tablet's replication back up to 3
  • equivalently, a node loses its disks and thus is essentially a new node
• add a new node to the cluster, and balance some existing data onto that one

Implementation-wise:

• ability to bootstrap a new node into a quorum (copy existing tablet data)
• ability to change-config a quorum to include a new peer
• some administrative commands to explicitly migrate data to perform manual load balancing (automatic load balancing tracked separately)

Issue Links

• blocks: KUDU-673 Automatic Master management of dynamic consensus configuration

Sub-Tasks

• Implement forced leader step-down: Resolved
• Implement RemoveServer RPC: Resolved
• Implement AddServer RPC: Resolved
• Implement a basic CLI tool to change a tablet quorum: Resolved
• Support remotely bootstrapping new nodes in tablet server: Resolved
• Support safely deleting (tombstoning) tablet replicas: Resolved
• Support replacement of tombstoned replicas on tablet server: Resolved
• Detect and replace failed replicas at the Master level: Resolved
• Allow tombstoned tablets to vote: Resolved
• Support PRE_VOTER config membership type: Resolved
• Tombstoned servers should reject remote bootstrap requests based on term of last log opid: Resolved
• Capability to delete & bootstrap followers that fall too far behind log: Resolved
• Re-replication support for Kudu beta: Resolved
• Higher availability re-replication support: Resolved
• Allow replicas that do not exist to vote: Resolved