Due to power failure event, we hit
We lost many racks across the cluster.
There were couple of missing blocks.
For a given missing block, following is the output of fsck.
After the power event, when we restarted the datanode, the blocks were in rbw directory.
When full block report is sent to namenode, all the blocks from rbw directory gets converted into RWR state and the namenode marked it as corrupt with reason Reason.INVALID_STATE.
After sometime (in this case after 31 hours) when I went to recover missing blocks, I noticed the following things.
All the datanodes has their copy of the block in rbw directory but the file was complete according to namenode.
All the replicas had the right size and correct genstamp and hdfs debug verify command also succeeded.
I went to dnA and moved the block from rbw directory to finalized directory.
Restarted the datanode (making sure the replicas file was not present during startup).
I forced a FBR and made sure the datanode block reported to namenode.
After waiting for sometime, still that block was missing.
I expected the missing block to go away since the replica is in FINALIZED directory.
On investigating more, I found out that namenode will remove the replica from corrupt map only if the reason for corruption was GENSTAMP_MISMATCH