[HDFS-3601] Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0, 2.1.0-beta
• Component/s: namenode
• Target Version/s: 2.0.5-alpha
• Hadoop Flags: Reviewed
• Assignee: Junping Du
• Reporter: Junping Du
• Created: 04/Jun/12 05:39
• Updated: 27/Aug/13 22:07
• Resolved: 03/Jun/13 05:00

Description

A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
1. The 3rd replica will be off node group of the 2nd replica
2. If there is no local node available, the 1st replica will be placed on a node in the local node group.

Issue Links

• depends upon: HDFS-3498 Make Replica Removal Policy pluggable and ReplicaPlacementPolicyDefault extensible for reusing code in subclass
• is depended upon by: HDFS-4240 In nodegroup-aware case, make sure nodes are avoided to place replica if some replica are already under the same nodegroup
• is depended upon by: YARN-19 4-layer topology (with NodeGroup layer) implementation of Container Assignment and Task Scheduling (for YARN)
• is part of: HADOOP-8468 Umbrella of enhancements to support different failure and locality topologies
• is related to: HDFS-3941 Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology