[HDDS-5686] Support transfer leadership to a specific follower master in raft based ha cluster

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Resolved
• Fix Version/s: 1.4.0
• Labels: pull-request-available
• Assignee: Xu Shao Hong
• Reporter: Xu Shao Hong
• Created: 27/Aug/21 02:24
• Updated: 21/Feb/23 02:22
• Resolved: 21/Feb/23 02:22

Description

Background:

In a raft based ha cluster, for some reason, we want to transfer leadership to a specific follower gracefully, now, we have to kill the leader master to transfer leadership, but the leadership could transfer to a unexpected follower, we have to kill the leader master again,this is so tricky.

 

Usage:

This feature is assumed to utilizing a shell command like ```ozone admin failover  -domain [SCM/OM]  -address [IpAddr] ```, with optional IpAddr we can transfer the leadership to the target follower.

 

Scenarios:

• We need to test every ratis node if each of them could be capable of being the leader.
• During replacing ratis nodes for some reason, we need to stabilize the process of changing the leadership, instead of forcely shuting down some node

 

Impl:

The alluxio community has already implemented this feature, we could backport from them.  They leverage the ratis support of setting priority for each member among the raft group which insures the highest priority node could become a leader.

Issue Links

• links to: GitHub Pull Request #2677
• links to: GitHub Pull Request #2686
• links to: GitHub Pull Request #4128

Sub-Tasks

• Support SCM transfer Ratis leadership: Resolved
• OM transition leadership: Resolved
• Reset RaftPeer priorities after transfer leadership: Resolved