[HBASE-24844] Exception on standalone (master) shutdown

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1
• Fix Version/s: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.6
• Component/s: Zookeeper
• Hadoop Flags: Reviewed
• Assignee: wenfeiyi666
• Reporter: Nick Dimiduk
• Created: 10/Aug/20 21:24
• Updated: 26/Aug/20 05:48
• Resolved: 12/Aug/20 11:51

Description

Running HBase (master branch) in standalone mode, terminating the process results in the following stack traces logged at error. It appears we shutdown the zookeeper client out-of-order with shutdown of the thread pools.

2020-08-10 14:21:46,777 INFO  [RS:0;localhost:16020] zookeeper.ZooKeeper: Session: 0x100111361f20001 closed
2020-08-10 14:21:46,778 INFO  [RS:0;localhost:16020] regionserver.HRegionServer: Exiting; stopping=localhost,16020,1597094491257; zookeeper connection closed.
2020-08-10 14:21:46,778 ERROR [main-EventThread] zookeeper.ClientCnxn: Error while calling watcher 
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6e61af4b rejected from java.util.concurrent.ThreadPoolExecutor@6a5e365f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
        at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
        at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:559)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2020-08-10 14:21:46,778 INFO  [shutdown-hook-0] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2020-08-10 14:21:46,779 ERROR [main-EventThread] zookeeper.ClientCnxn: Error while calling watcher 
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7d41da91 rejected from java.util.concurrent.ThreadPoolExecutor@6a5e365f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
        at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
        at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:559)
        at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2020-08-10 14:21:46,780 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x100111361f20001
2020-08-10 14:21:46,780 INFO  [shutdown-hook-0] regionserver.ShutdownHook: Shutdown hook finished.

Issue Links

• links to: GitHub Pull Request #2239