[HBASE-11594] Unhandled NoNodeException in distributed log replay mode

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.99.0, 0.98.5
• Hadoop Flags: Reviewed
• Assignee: Jeffrey Zhong
• Reporter: Jeffrey Zhong
• Created: 26/Jul/14 00:12
• Updated: 06/Apr/18 17:51
• Resolved: 01/Aug/14 16:40

Description

This issue happens when a RS doesn't have any WAL to be replayed. Master immediately finishes recovery for the RS while a region in recovery is still opening.

Below is the exception in region server log:

org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/recovering-regions/20fcfad9746b3d83fff84fb773af6c80/h2-suse-uns-1395117052-hbase-7.cs1cloud.internal,60020,1395141895633
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1266)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:407)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:878)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:928)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:922)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.updateRecoveringRegionLastFlushedSequenceId(HRegionServer.java:4560)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1780)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:325)

The following is related master log:

2014-03-18 11:27:14,192 DEBUG [MASTER_SERVER_OPERATIONS-h2-suse-uns-1395117052-hbase-4:60000-3] master.DeadServer: Finished processing h2-suse-uns-1395117052-hbase-7.cs1cloud.internal,60020,1395141895633
2014-03-18 11:27:14,199 INFO  [M_LOG_REPLAY_OPS-h2-suse-uns-1395117052-hbase-4:60000-0] master.MasterFileSystem: Log dir for server h2-suse-uns-1395117052-hbase-7.cs1cloud.internal,60020,1395141895633 does not exist
2014-03-18 11:27:14,203 INFO  [M_LOG_REPLAY_OPS-h2-suse-uns-1395117052-hbase-4:60000-0] master.SplitLogManager: dead splitlog workers [h2-suse-uns-1395117052-hbase-7.cs1cloud.internal,60020,1395141895633]
2014-03-18 11:27:14,204 DEBUG [M_LOG_REPLAY_OPS-h2-suse-uns-1395117052-hbase-4:60000-0] master.SplitLogManager: Scheduling batch of logs to split
2014-03-18 11:27:14,206 INFO  [M_LOG_REPLAY_OPS-h2-suse-uns-1395117052-hbase-4:60000-0] master.SplitLogManager: started splitting 0 logs in []