[PHOENIX-3260] MetadataRegionObserver.postOpen() can prevent region server from shutting down for a long duration

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.8.1
• Assignee: Samarth Jain
• Reporter: Samarth Jain
• Created: 09/Sep/16 17:16
• Updated: 28/Sep/16 05:15
• Resolved: 09/Sep/16 18:24

Description

On my local setup, I noticed that invoking ./stop-hbase.sh wasn't able to shutdown the hbase cluster for a long time. Doing a jstack revealed that the thread being created in MetadataRegionObserver.postOpen() is not assigned as daemon. This could prevent the region server hosting the SYSTEM.CATALOG table from shutting down for a long duration.

Stacktrace of the hung thread:

"Thread-87" prio=5 tid=0x00007ffbd699d800 nid=0x1b607 waiting on condition [0x0000000120446000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:149)
	- locked <0x00000007d1ab8140> (a org.apache.hadoop.hbase.client.RpcRetryingCaller)
	at org.apache.hadoop.hbase.client.HTable.get(HTable.java:833)
	at org.apache.hadoop.hbase.client.HTable.get(HTable.java:810)
	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment$HTableWrapper.get(CoprocessorHost.java:466)
	at org.apache.phoenix.util.UpgradeUtil.truncateStats(UpgradeUtil.java:1226)
	at org.apache.phoenix.coprocessor.MetaDataRegionObserver$1.run(MetaDataRegionObserver.java:148)
	at java.lang.Thread.run(Thread.java:745)