[HDDS-698] Support Topology Awareness for Ozone

• Type: New Feature
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: SCM
• Assignee: Sammi Chen
• Reporter: Xiaoyu Yao
• Created: 19/Oct/18 04:55
• Updated: 12/Aug/20 14:02
• Resolved: 12/Aug/20 14:02

Description

This is an umbrella JIRA to add topology aware support for Ozone Pipelines, Containers and Blocks. Long time since HDFS is created, we provide rack/nodegroup awareness for reliability and high performance for data access. Ozone need a similar mechanism and this can be more flexible for cloud scenarios.

Issue Links

• is a parent of: HDDS-3722 Network Topology improvements post 0.6.0
• relates to: HDDS-1564 Ozone multi-raft support
• links to: Design Doc

Sub-Tasks

• Detect Ozone Network topology: Resolved
• Support YAML format network topology cluster definition: Resolved
• Support rack awared node placement policy based on network topology: Resolved
• Fix random test failure TestSCMContainerPlacementRackAware: Resolved
• Add "scmcli printTopology" shell command to print datanode topology: Resolved
• Support configurable container placement policy through "ozone.scm.container.placement.classname": Resolved
• Missing test resources of integrataion-test project in target directory after compile: Resolved
• Add datanode to network topology cluster during node register: Resolved
• Allow Ozone RPC client to read with topology awareness: Resolved
• NPE thrown while trying to find DN closest to client: Resolved
• Make Topology Aware Replication/Read non-default for ozone 0.4.1: Resolved
• Add option to "ozone scmcli printTopology" to order the output acccording to topology layer: Resolved
• ReplicationManager fail to find proper node topology based on Datanode details from heartbeat: Resolved
• Change topology sorting related logs in Pipeline from INFO to DEBUG: Resolved
• Turn on topology aware read in TestFailureHandlingByClient: Resolved
• Suppress WARN log from NetworkTopology#getDistanceCost: Resolved
• Use "ozone.network.topology.aware.read" to control both RPC client and server side logic: Resolved
• TestKeyManagerImpl.testLookupKeyWithLocation is failing: Resolved
• Support listPipelines by filters in scmcli: Resolved
• Add metrics in rack aware container placement policy: Resolved
• Support multiple excluded scopes when choosing datanodes in NetworkTopology: Resolved
• Choose datanode for pipeline creation based on network topology: Resolved
• Replication manager should detect and correct containers which don't meet the replication policy: Resolved
• Refactor recon missing containers task to detect under, over and mis-replicated containers: Resolved
• Extend network topology acceptance test to read data when datanodes are stopped: Resolved
• Enable test added in HDDS-3084 when blocking issues are resolved: Resolved
• Test Topology Aware Job scheduling with Ozone Topology: Resolved