[HBASE-27426] Region server abort with failing to list region servers.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0, 2.5.1, 3.0.0-alpha-4, 2.4.16
• Component/s: Zookeeper
• Hadoop Flags: Reviewed
• Assignee: Victor Li
• Reporter: Victor Li
• Created: 12/Oct/22 23:11
• Updated: 25/Apr/23 15:50
• Resolved: 17/Oct/22 21:31

Description

During unit tests, clusters were created and destroyed, the region server could abort during shutdown with failing to list region servers.

 

Logs:

 

2022-10-11 18:44:43,393 INFO  [main] utility.HBaseClusterSetup(160): Starting mini cluster: singleCluster

[master/victorli-wsl:0:becomeActiveMaster] INFO org.apache.hadoop.hbase.zookeeper.ZKWatcher - not a secure deployment, proceeding

2022-10-11 18:45:07,234 INFO  [Listener at localhost/45185] utility.HBaseClusterSetup(177): Started mini cluster: singleCluster

[Listener at localhost/45185] INFO org.apache.hadoop.hbase.zookeeper.ZKWatcher - ZKWatcher close started.

[Listener at localhost/45185] INFO org.apache.hadoop.hbase.zookeeper.ZKWatcher - ZKWatcher close finished.

[Listener at localhost/45185] INFO org.apache.hadoop.hbase.zookeeper.ZKWatcher - ZKWatcher close started.

[Listener at localhost/45185] INFO org.apache.hadoop.hbase.zookeeper.ZKWatcher - ZKWatcher close finished.

2022-10-11 18:45:45,980 INFO  [Listener at localhost/45185] utility.HBaseClusterSetup(132): Stopping mini cluster: singleCluster

[RS:0;victorli-wsl:35113] INFO org.apache.hadoop.hbase.zookeeper.ZKWatcher - ZKWatcher close started.

[zk-event-processor-pool-0] ERROR org.apache.hadoop.hbase.regionserver.HRegionServer - ***** ABORTING region server victorli-wsl,35113,1665539090704: failed to list region servers *****

[RS:0;victorli-wsl:35113] INFO org.apache.hadoop.hbase.zookeeper.ZKWatcher - ZKWatcher close finished.

[M:0;victorli-wsl:42915] INFO org.apache.hadoop.hbase.zookeeper.ZKWatcher - ZKWatcher close started.

[M:0;victorli-wsl:42915] INFO org.apache.hadoop.hbase.zookeeper.ZKWatcher - ZKWatcher close finished.

2022-10-11 18:45:57,208 INFO  [Listener at localhost/45185] utility.HBaseClusterSetup(144): Stopped mini cluster: singleCluster

 

Stack:

[zk-event-processor-pool-0] WARN org.apache.hadoop.hbase.zookeeper.ZKUtil - regionserver:35113-0x101b6fed2a00001, quorum=127.0.0.1:53028, baseZNode=/9 Unable to list children of znode /9/rs

java.lang.InterruptedException

    at java.lang.Object.wait(Native Method)

    at java.lang.Object.wait(Object.java:502)

    at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1601)

    at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1573)

    at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2420)

    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:341)

    at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:190)

    at org.apache.hadoop.hbase.zookeeper.RegionServerAddressTracker.loadRegionServerList(RegionServerAddressTracker.java:54)

    at org.apache.hadoop.hbase.zookeeper.RegionServerAddressTracker.nodeChildrenChanged(RegionServerAddressTracker.java:71)

    at org.apache.hadoop.hbase.zookeeper.ZKWatcher.lambda$processEvent$0(ZKWatcher.java:586)

    at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187)

    at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:177)

    at org.apache.hadoop.hbase.zookeeper.ZKWatcher.processEvent(ZKWatcher.java:554)

    at org.apache.hadoop.hbase.zookeeper.ZKWatcher.lambda$process$1(ZKWatcher.java:607)

    at org.apache.hadoop.hbase.trace.TraceUtil.lambda$tracedRunnable$2(TraceUtil.java:155)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

    at java.lang.Thread.run(Thread.java:750)

Issue Links

• causes: HBASE-27810 HBCK throws RejectedExecutionException when closing ZooKeeper resources
• links to: GitHub Pull Request #4829