[HBASE-23364] HRegionServer sometimes does not shut down.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 2.3.0, 1.6.0
• Fix Version/s: 3.0.0-alpha-1, 2.3.0, 1.6.0
• Component/s: metrics, regionserver
• Assignee: Lars Hofhansl
• Reporter: Lars Hofhansl
• Created: 20/Nov/19 07:25
• Updated: 07/Dec/19 00:01
• Resolved: 06/Dec/19 16:52

Description

Note that I initially assumed this to be a Phoenix bug. But I tracked it down to HBase.

---

I noticed that recently only. Latest build from HBase's branch-1 and latest build from Phoenix' 4.x-HBase-1.5. I don't know, yet, whether it's a Phoenix or an HBase issues.

Just filing it here for later reference.

jstack show this thread as the only non-daemon thread:

"pool-11-thread-1" #470 prio=5 os_prio=0 tid=0x0000558a709a4800 nid=0x238e waiting on condition [0x00007f213ad68000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000058eafece8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

No other information. Somebody created a thread pool somewhere and forgot to set the threads to daemon or is not shutting down the pool properly.

Edit: I looked for other reference of the locked objects in the stack dump, but didn't find any.

 

 

Issue Links

• is broken by: HBASE-15519 Add per-user metrics
• is related to: HBASE-23210 Backport HBASE-15519 (Add per-user metrics) to branch-1