[HADOOP-8468] Umbrella of enhancements to support different failure and locality topologies

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0, 2.0.0-alpha
• Component/s: ha, io
• Target Version/s: 2.1.0-beta
• Assignee: Junping Du
• Reporter: Junping Du
• Created: 04/Jun/12 04:08
• Updated: 22/Feb/19 10:44
• Resolved: 22/Feb/19 10:44

Description

The current hadoop network topology (described in some previous issues like: Hadoop-692) works well in classic three-tiers network when it comes out. However, it does not take into account other failure models or changes in the infrastructure that can affect network bandwidth efficiency like: virtualization.
Virtualized platform has following genes that shouldn't been ignored by hadoop topology in scheduling tasks, placing replica, do balancing or fetching block for reading:
1. VMs on the same physical host are affected by the same hardware failure. In order to match the reliability of a physical deployment, replication of data across two virtual machines on the same host should be avoided.
2. The network between VMs on the same physical host has higher throughput and lower latency and does not consume any physical switch bandwidth.
Thus, we propose to make hadoop network topology extend-able and introduce a new level in the hierarchical topology, a node group level, which maps well onto an infrastructure that is based on a virtualized environment.

Issue Links

• incorporates: HDFS-4240 In nodegroup-aware case, make sure nodes are avoided to place replica if some replica are already under the same nodegroup
• incorporates: YARN-18 Configurable Hierarchical Topology for YARN
• incorporates: YARN-19 4-layer topology (with NodeGroup layer) implementation of Container Assignment and Task Scheduling (for YARN)
• incorporates: HDFS-3495 Update Balancer to support new NetworkTopology with NodeGroup
• incorporates: HDFS-3601 Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
• incorporates: HDFS-3498 Make Replica Removal Policy pluggable and ReplicaPlacementPolicyDefault extensible for reusing code in subclass
• is related to: HDFS-4886 Override verifyBlockPlacement() API in BlockPlacementPolicyWithNodeGroup
• is related to: HDFS-4898 BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to properly fallback to local rack
• relates to: HADOOP-11326 documentation for configuring HVE: dfs.block.replicator.classname should be org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyWithNodeGroup
• relates to: HDFS-3564 Design enhancements to the pluggable blockplacementpolicy

Sub-Tasks

• Make NetworkTopology class pluggable: Closed
• Implementation of 4-layer subclass of NetworkTopology (NetworkTopologyWithNodeGroup): Closed
• Backport Network Topology Extension for Virtualization (HADOOP-8468) to branch-1: Closed
• Document usage of node-group layer topology: Resolved
• CLONE - Backport Network Topology Extension for Virtualization (HADOOP-8468) to branch-1: Resolved