[HDFS-15] Rack replication policy can be violated for over replicated blocks

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.20.3
• Fix Version/s: 0.20.3, 0.21.0
• Assignee: Jitendra Nath Pandey
• Reporter: Hairong Kuang
• Created: 21/Oct/08 18:01
• Updated: 12/Apr/12 06:27
• Resolved: 31/Aug/09 17:32

Description

HDFS replicas placement strategy guarantees that the replicas of a block exist on at least two racks when its replication factor is greater than one. But fsck still reports that the replicas of some blocks end up on one rack.

The cause of the problem is that decommission and corruption handling only check the block's replication factor but not the rack requirement. When an over-replicated block loses a replica due to decomission, corruption, or heartbeat lost, namenode does not take any action to guarantee that remaining replicas are on different racks.

Issue Links

• is related to: HDFS-37 An invalidated block should be removed from the blockMap
• is related to: HDFS-1480 All replicas of a block can end up on the same rack when some datanodes are decommissioning.
• is related to: HDFS-3256 HDFS considers blocks under-replicated if topology script is configured with only 1 rack
• is related to: HDFS-1562 Add rack policy tests
• relates to: HADOOP-4489 Blocks have been scheduled to delete by NameNode but are not deleted on the DataNode