[HBASE-14370] Use separate thread for calling ZKPermissionWatcher#refreshNodes()

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.98.0
• Fix Version/s: 1.3.0, 0.98.15, 2.0.0
• Hadoop Flags: Reviewed
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 05/Sep/15 16:15
• Updated: 01/Jul/22 21:14
• Resolved: 26/Sep/15 01:25

Description

I came off a support case (0.98.0) where main zk thread was seen doing the following:

  at org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.refreshAuthManager(ZKPermissionWatcher.java:152)
  at org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.refreshNodes(ZKPermissionWatcher.java:135)
  at org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.nodeChildrenChanged(ZKPermissionWatcher.java:121)
  at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348)
  at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
  at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)

There were 62000 nodes under /acl due to lack of fix from HBASE-12635, leading to slowness in table creation because zk notification for region offline was blocked by the above.

The attached patch separates refreshNodes() call into its own thread.

Thanks to Enis and Devaraj for offline discussion.

Issue Links

• relates to: HBASE-18628 ZKPermissionWatcher blocks all ZK notifications