[HBASE-5549] Master can fail if ZooKeeper session expires

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.95.2
• Fix Version/s: 0.94.2
• Component/s: master, Zookeeper

Environment

all

• Hadoop Flags: Reviewed
• Assignee: Nicolas Liochon
• Reporter: Nicolas Liochon
• Created: 09/Mar/12 12:19
• Updated: 07/Apr/13 04:34
• Resolved: 16/Mar/12 21:40

Description

There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.

This can happen in real life, it can happen when:

• master & zookeeper starts
• zookeeper connection is cut
• master enters the retry loop
• in the meantime the session expires
• the network comes back, the session is recreated
• the retries continues, but on the wrong object, hence fails.

Issue Links

• is related to: HBASE-5399 Cut the link between the client and the zookeeper ensemble
• is related to: HBASE-5572 KeeperException.SessionExpiredException management could be improved in Master
• supercedes: HBASE-5572 KeeperException.SessionExpiredException management could be improved in Master