[HBASE-5681] Split Region crash if region is still offline after a previous split

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 0.92.1, 0.94.1, 0.95.2
• Component/s: regionserver, Zookeeper
• Assignee: Unassigned
• Reporter: Matteo Bertozzi
• Created: 30/Mar/12 19:29
• Updated: 13/Jun/22 16:01
• Resolved: 01/Apr/12 20:45

Description

I've a script that starts hbase and a couple of region servers in distributed mode (hbase.cluster.distributed = true) due to HBASE-5666 I need a sleep to ensure that rs are up.

$HBASE_HOME/bin/start-hbase.sh
sleep 5 # bug HBASE-5666 rs doesn't retry if znode is not available.
$HBASE_HOME/bin/local-regionservers.sh start 1 2 3

Once hbase is started I run an hbase shell script file (see below)
everything is fine till last split operation.

# $HBASE_HOME/bin/hbase shell test.hbase
# test.hbase
create 'bugtb-t1', 'tcf11', 'tcf12'
create 'bugtb-t2', 'tcf11', 'tcf12'

put 'bugtb-t1', '10', 'tcf11:c1', 'a'
put 'bugtb-t1', '15', 'tcf11:c2', 'b'
put 'bugtb-t1', '20', 'tcf11:c1', 'c'
put 'bugtb-t1', '30', 'tcf11:c2', 'd'
put 'bugtb-t1', '35', 'tcf11:c1', 'e'
put 'bugtb-t1', '40', 'tcf11:c2', 'f'

put 'bugtb-t2', '10', 'tcf11:c1', 'a'
put 'bugtb-t2', '15', 'tcf11:c2', 'b'
put 'bugtb-t2', '20', 'tcf11:c1', 'c'
put 'bugtb-t2', '30', 'tcf11:c2', 'd'
put 'bugtb-t2', '35', 'tcf11:c1', 'e'
put 'bugtb-t2', '40', 'tcf11:c2', 'f'

split 'bugtb-t1', '20'
split 'bugtb-t2', '20'
split 'bugtb-t1', '40'

During the last split the region is still offline, and you get an exception....
(If you sleep a bit before executing the last split, everything is fine)

ERROR: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: bugtb-t1,,1333134892936.4e14c2cf4293156d5b099dc3d5c44890.
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3123)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2926)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1383)