[HBASE-10335] AuthFailedException in zookeeper may block replication forever

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.94.15, 0.99.0
• Fix Version/s: 0.98.0, 0.96.2, 0.99.0, 0.94.17
• Component/s: Replication, security
• Hadoop Flags: Reviewed
• Assignee: Shaohui Liu
• Reporter: Shaohui Liu
• Created: 14/Jan/14 08:48
• Updated: 26/Feb/14 04:45
• Resolved: 15/Jan/14 06:24

Description

ReplicationSource will rechoose sinks when encounted exceptions during skipping edits to the current sink. But if the zookeeper client for peer cluster go to AUTH_FAILED state, the ReplicationSource will always get AuthFailedException. The ReplicationSource does not reconnect the peer, because reconnectPeer only handle ConnectionLossException and SessionExpiredException. As a result, the replication will print log:

2014-01-14,12:07:06,892 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 0 rs from peer cluster # 20
2014-01-14,12:07:06,892 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Slave cluster looks down: 20 has 0 region servers

and be blocked forever.

I think other places may have same problems for not handling AuthFailedException in zookeeper. eg: HBASE-8675.
apurtell