[HBASE-9387] Region could get lost during assignment

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.95.2
• Fix Version/s: 0.98.0, 0.96.0
• Component/s: Region Assignment
• Hadoop Flags: Reviewed
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 29/Aug/13 23:30
• Updated: 20/Nov/15 11:55
• Resolved: 05/Sep/13 00:48

Description

I observed test timeout running against hadoop 2.1.0 with distributed log replay turned on.
Looks like region state for 1588230740 became inconsistent between master and the surviving region server:

2013-08-29 22:15:34,180 INFO  [AM.ZK.Worker-pool2-t4] master.RegionStates(299): Onlined 1588230740 on kiyo.gq1.ygridcore.net,57016,1377814510039
...
2013-08-29 22:15:34,587 DEBUG [Thread-221] client.HConnectionManager$HConnectionImplementation(1269): locateRegionInMeta parentTable=hbase:meta, metaLocation={region=hbase:meta,,1.1588230740, hostname=kiyo.gq1.ygridcore.net,57016,1377814510039, seqNum=0}, attempt=2 of 35 failed; retrying after sleep of 302 because: org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region is being opened: 1588230740
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2574)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3949)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2733)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26965)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2063)
        at org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1800)
        at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
        at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)