[HDFS-3941] Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.2.0, 1-win
• Component/s: namenode
• Target Version/s: 1.2.0
• Hadoop Flags: Reviewed
• Assignee: Junping Du
• Reporter: Junping Du
• Created: 16/Sep/12 19:06
• Updated: 15/May/13 05:16
• Resolved: 17/Nov/12 03:50

Description

With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
0. No more than one replica is placed within a NodeGroup
1. First replica on the local node.
2. Second and third replicas are within the same rack but remote rack with 1st replica.
3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.

Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.

Issue Links

• is blocked by: HADOOP-8820 Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
• Is contained by: HADOOP-8817 Backport Network Topology Extension for Virtualization (HADOOP-8468) to branch-1
• relates to: HDFS-3601 Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
• relates to: HDFS-3498 Make Replica Removal Policy pluggable and ReplicaPlacementPolicyDefault extensible for reusing code in subclass