[HDFS-385] Design a pluggable interface to place replicas of blocks in HDFS

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.21.0, 1.2.0, 1-win
• Hadoop Flags: Reviewed
• Release Note: New experimental API BlockPlacementPolicy allows investigating alternate rules for locating block replicas.
• Assignee: Dhruba Borthakur
• Reporter: Dhruba Borthakur
• Created: 21/Jul/08 12:18
• Updated: 30/Jul/14 21:42
• Resolved: 15/Sep/09 09:40

Description

The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models.

Issue Links

• is related to: HDFS-347 DFS read performance suboptimal when client co-located on nodes with data
• relates to: HDFS-629 Remove ReplicationTargetChooser.java along with fixing import warnings.
• relates to: HDFS-1094 Intelligent block placement policy to decrease probability of block loss
• relates to: HDFS-1451 Extend the pluggable block placement policy interface to handle/allow file-level intelligence
• relates to: HDFS-3564 Design enhancements to the pluggable blockplacementpolicy
• relates to: HDFS-3649 Port HDFS-385 to branch-1-win