[HDFS-900] Corrupt replicas are not tracked correctly through block report from DN

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.22.0
• Fix Version/s: 0.22.0
• Hadoop Flags: Reviewed
• Assignee: Konstantin Shvachko
• Reporter: Todd Lipcon
• Created: 14/Jan/10 00:41
• Updated: 12/Dec/11 06:19
• Resolved: 04/Feb/11 04:12

Description

This one is tough to describe, but essentially the following order of events is seen to occur:

1. A client marks one replica of a block to be corrupt by telling the NN about it
2. Replication is then scheduled to make a new replica of this node
3. The replication completes, such that there are now 3 good replicas and 1 corrupt replica
4. The DN holding the corrupt replica sends a block report. Rather than telling this DN to delete the node, the NN instead marks this as a new good replica of the block, and schedules deletion on one of the good replicas.

I don't know if this is a dataloss bug in the case of 1 corrupt replica with dfs.replication=2, but it seems feasible. I will attach a debug log with some commentary marked by '============>', plus a unit test patch which I can get to reproduce this behavior reliably. (it's not a proper unit test, just some edits to an existing one to show it)

Issue Links

• relates to: HDFS-875 NameNode incorretly handles corrupt replicas