[HADOOP-8817] Backport Network Topology Extension for Virtualization (HADOOP-8468) to branch-1

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.2.0
• Labels: features
• Target Version/s: 1.2.0
• Release Note: A new 4-layer network topology NetworkToplogyWithNodeGroup is available to make Hadoop more robust and efficient in virtualized environment.
• Assignee: Junping Du
• Reporter: Junping Du
• Created: 16/Sep/12 15:29
• Updated: 30/Jul/15 15:25
• Resolved: 19/Mar/13 17:41

Description

HADOOP-8468 propose network topology changes for running on virtualized infrastructure, which includes:
1. Add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup): HADOOP-8469, HADOOP-8470
2. Update Replica Placement/Removal Policy to reflect new topology layer: HDFS-3498, HDFS-3601
3. Update balancer policy:HDFS-3495
4. Update Task Scheduling Policy to reflect new topology layer and support the case that compute nodes (NodeManager or TaskTracker) and data nodes are separated into different VMs, but still benefit from physical host locality: YARN-18, YARN-19.
This JIRA will address the backport work on branch-1 which will be divided into 4 issues/patches in related jira issues.

Issue Links

• contains: HADOOP-8820 Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
• contains: HDFS-3941 Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
• contains: HDFS-3942 Backport HDFS-3495: Update balancer policy for Network Topology with additional 'NodeGroup' layer
• contains: MAPREDUCE-4660 Update task placement policy for NetworkTopology with 'NodeGroup' layer
• is duplicated by: HADOOP-12283 CLONE - Backport Network Topology Extension for Virtualization (HADOOP-8468) to branch-1