[HBASE-10345] HMaster should not serve when disconnected with ZooKeeper

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Later
• Affects Version/s: 0.94.3
• Component/s: master
• Assignee: Unassigned
• Reporter: chendihao
• Created: 15/Jan/14 09:46
• Updated: 17/Jun/22 04:59
• Resolved: 05/Jun/21 15:01

Description

Refer to HBASE-9468(Previous active master can still serves RPC request when it is trying recovering expired zk session), we can fail fast to avoid existing double masters at the same time. But this problem may occur before session expired. When receive Disconnected event, we can't make sure of that this active master can communicate with zk later. And it doesn't know whether backup master has become the new active master or not until it receives Expired event(which may lose forever). During this unsure-who-is-active-master period, the current active master should not serve(maybe turn off RpcServer).

Here is the statement from "ZooKeeper Distributed Process Coordination" P101

If the developer is not careful, the old leader will continue to act as a leader and may take actions that conflict with those of the new leader. For this reason, when a process receives a Disconnected event, the process should suspend actions taken as a leader until it reconnects. Normally this reconnect happens very quickly.

So it's equally necessary to handle Disconnected event and Expired event.