[HBASE-8490] Region mover script can get stuck in infinite loop if UnknownRegionException

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 0.92.1
• Component/s: shell

Environment

HBase 0.92.1-cdh4.1.3

• Assignee: Unassigned
• Reporter: Esteban Gutierrez
• Created: 03/May/13 18:12
• Updated: 16/Jun/22 17:03
• Resolved: 29/Apr/14 00:28

Description

13/05/02 23:56:55 INFO region_mover: Moving 1 region(s) from procyon-8.nova.cloudera.com,60020,1367530634629 during this cycle
13/05/02 23:56:55 INFO region_mover: Moving region 588ccd887c699c7c23188a79a3d1807f (0 of 1) to server=procyon-5.nova.cloudera.com,60020,1367542172830
13/05/02 23:56:55 INFO region_mover: Exception moving 588ccd887c699c7c23188a79a3d1807f; split/moved? Continuing: java.lang.reflect.UndeclaredThrowableException: org.apache.hadoop.hbase.UnknownRegionException: 588ccd887c699c7c23188a79a3d1807f
	at org.apache.hadoop.hbase.master.HMaster.move(HMaster.java:1015)
	at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)

This was triggered due IllegalStateException while trying to open an unexisting region:

2013-05-02 16:23:06,558 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: TestTable,,1367530404417.588ccd887c699c7c23188a79a3d1
807f.
2013-05-02 16:23:06,583 INFO org.apache.hadoop.hbase.util.FSUtils: hdfs://nameservice1/hbase/TestTable doesn't exist
2013-05-02 16:23:06,586 INFO org.apache.hadoop.hbase.util.FSUtils: hdfs://nameservice1/hbase/TestTable doesn't exist
2013-05-02 16:23:06,586 WARN org.apache.hadoop.hbase.util.FSTableDescriptors: The following folder is in HBase's root directory and doesn't contain a table descriptor,
 do consider deleting it: TestTable
2013-05-02 16:23:06,666 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=TestTable,,1367530404417.588ccd887c699c7c23188a79a3
d1807f.
java.lang.IllegalStateException: Could not instantiate a region instance.
	at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3123)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3254)
	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
	at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3120)
	... 7 more
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:133)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:125)
	at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:376)
	... 12 more

Issue Links

• duplicates: HBASE-5356 region_mover.rb can hang if table region it belongs to is deleted.