[HBASE-8912] [0.94] AssignmentManager throws IllegalStateException from PENDING_OPEN to OFFLINE

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.94.16
• Hadoop Flags: Reviewed
• Assignee: Lars Hofhansl
• Reporter: Enis Soztutar
• Created: 10/Jul/13 01:21
• Updated: 21/Jan/14 01:31
• Resolved: 02/Jan/14 21:52

Description

AM throws this exception which subsequently causes the master to abort:

java.lang.IllegalStateException: Unexpected state : testRetrying,jjj,1372891751115.9b828792311001062a5ff4b1038fe33b. state=PENDING_OPEN, ts=1372891751912, server=hemera.apache.org,39064,1372891746132 .. Cannot transit it to OFFLINE.
	at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
	at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
	at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
	at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
	at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
	at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

This exception trace is from the failing test TestMetaReaderEditor which is failing pretty frequently, but looking at the test code, I think this is not a test-only issue, but affects the main code path.

https://builds.apache.org/job/HBase-0.94/1036/testReport/junit/org.apache.hadoop.hbase.catalog/TestMetaReaderEditor/testRetrying/