[HBASE-14968] ConcurrentModificationException in region close resulting in the region staying in closing state

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17, 2.0.0
• Component/s: Region Assignment, regionserver
• Hadoop Flags: Reviewed
• Assignee: Enis Soztutar
• Reporter: Enis Soztutar
• Created: 12/Dec/15 02:38
• Updated: 04/Aug/16 17:20
• Resolved: 15/Dec/15 19:52

Description

We have seen this in our tests. The region gets closed, but the region close handler gets an unexpected exception causing the region to stay in closing state until RS is restarted.

The Phoenix and security coprocessors were loaded in that region. Here is the stack trace:

java.util.ConcurrentModificationException
	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
	at java.util.ArrayList$Itr.next(ArrayList.java:851)
	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.shutdown(CoprocessorHost.java:442)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment.shutdown(RegionCoprocessorHost.java:155)
	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.shutdown(CoprocessorHost.java:272)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$5.postEnvCall(RegionCoprocessorHost.java:496)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1761)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postClose(RegionCoprocessorHost.java:489)
	at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1502)
	at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1349)
	at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:138)
	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Issue Links

• relates to: HBASE-16350 Undo server abort from HBASE-14968