[HDFS-5012] replica.getGenerationStamp() may be >= recoveryId

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 2.0.5-alpha
• Assignee: Unassigned
• Reporter: Ted Yu
• Created: 19/Jul/13 04:30
• Updated: 07/Nov/17 01:32
• Resolved: 05/Nov/17 03:36

Description

The following was first observed by jdcryans in TestReplicationQueueFailover running against 2.0.5-alpha:

2013-07-16 17:14:33,340 ERROR [IPC Server handler 7 on 35081] security.UserGroupInformation(1481): PriviledgedActionException as:ec2-user (auth:SIMPLE) cause:java.io.IOException: THIS IS NOT SUPPOSED TO HAPPEN: replica.getGenerationStamp() >= recoveryId = 1041, block=blk_4297992342878601848_1041, replica=FinalizedReplica, blk_4297992342878601848_1041, FINALIZED
  getNumBytes()     = 794
  getBytesOnDisk()  = 794
  getVisibleLength()= 794
  getVolume()       = /home/ec2-user/jenkins/workspace/HBase-0.95-Hadoop-2/hbase-server/target/test-data/f2763e32-fe49-4988-ac94-eeca82431821/dfscluster_643a635e-4e39-4aa5-974c-25e01db16ff7/dfs/data/data3/current
  getBlockFile()    = /home/ec2-user/jenkins/workspace/HBase-0.95-Hadoop-2/hbase-server/target/test-data/f2763e32-fe49-4988-ac94-eeca82431821/dfscluster_643a635e-4e39-4aa5-974c-25e01db16ff7/dfs/data/data3/current/BP-1477359609-10.197.55.49-1373994849464/current/finalized/blk_4297992342878601848
  unlinked          =false
2013-07-16 17:14:33,341 WARN  [org.apache.hadoop.hdfs.server.datanode.DataNode$2@64a1fcba] datanode.DataNode(1894): Failed to obtain replica info for block (=BP-1477359609-10.197.55.49-1373994849464:blk_4297992342878601848_1041) from datanode (=127.0.0.1:47006)
java.io.IOException: THIS IS NOT SUPPOSED TO HAPPEN: replica.getGenerationStamp() >= recoveryId = 1041, block=blk_4297992342878601848_1041, replica=FinalizedReplica, blk_4297992342878601848_1041, FINALIZED
  getNumBytes()     = 794
  getBytesOnDisk()  = 794
  getVisibleLength()= 794
  getVolume()       = /home/ec2-user/jenkins/workspace/HBase-0.95-Hadoop-2/hbase-server/target/test-data/f2763e32-fe49-4988-ac94-eeca82431821/dfscluster_643a635e-4e39-4aa5-974c-25e01db16ff7/dfs/data/data3/current
  getBlockFile()    = /home/ec2-user/jenkins/workspace/HBase-0.95-Hadoop-2/hbase-server/target/test-data/f2763e32-fe49-4988-ac94-eeca82431821/dfscluster_643a635e-4e39-4aa5-974c-25e01db16ff7/dfs/data/data3/current/BP-1477359609-10.197.55.49-1373994849464/current/finalized/blk_4297992342878601848
  unlinked          =false

Issue Links

• duplicates: HDFS-2970 Contending block synchronizations can result in scary log messages
• is related to: HDFS-11576 Block recovery will fail indefinitely if recovery time > heartbeat interval