[HDFS-13418] NetworkTopology should be configurable when enable DFSNetworkTopology

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 2.10.0, 3.2.0, 3.1.1
• Target Version/s: 2.10.0, 3.2.0, 3.1.1
• Hadoop Flags: Reviewed
• Assignee: Tao Jie
• Reporter: Tao Jie
• Created: 10/Apr/18 02:36
• Updated: 13/Apr/18 10:25
• Resolved: 13/Apr/18 10:01

Description

In HDFS-11530 we introduce DFSNetworkTopology and in HDFS-11998 we set DFSNetworkTopology as the default implementation.

We still have net.topology.impl=org.apache.hadoop.net.NetworkTopology in core-site.default. Actually this property does not effect once dfs.use.dfs.network.topology is true.
in DatanodeManager，networkTopology is initialized as

    if (useDfsNetworkTopology) {
      networktopology = DFSNetworkTopology.getInstance(conf);
    } else {
      networktopology = NetworkTopology.getInstance(conf);
    }

I think we should still make the NetworkTopology configurable rather than hard code the implementation since we may need another NetworkTopology impl.
I am not sure if there is other consideration. Any thought? vagarychen linyiqun

Issue Links

• relates to: HDFS-11530 Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
• relates to: HDFS-11998 Enable DFSNetworkTopology as default