[HDFS-16517] In 2.10 the distance metric is wrong for non-DN machines

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.10.1
• Fix Version/s: 2.10.2
• Labels: pull-request-available
• External issue URL: https://github.com/apache/hadoop/pull/4091
• Assignee: Owen O'Malley
• Reporter: Owen O'Malley
• Created: 22/Mar/22 00:08
• Updated: 23/Mar/22 22:10
• Resolved: 23/Mar/22 22:10

Description

In 2.10, the metric for distance between the client and the data node is wrong for machines that aren't running data nodes (ie. getWeightUsingNetworkLocation). The code works correctly in 3.3+.

Currently

 

Client	DataNode	getWeight	getWeightUsingNetworkLocation
/rack1/node1	/rack1/node1	0	0
/rack1/node1	/rack1/node2	2	2
/rack1/node1	/rack2/node2	4	2
/pod1/rack1/node1	/pod1/rack1/node2	2	2
/pod1/rack1/node1	/pod1/rack2/node2	4	2
/pod1/rack1/node1	/pod2/rack2/node2	6	4

 

This bug will destroy data locality on clusters where the clients share racks with DataNodes, but are running on machines that aren't running DataNodes, such as striping federated HDFS clusters across racks.

Issue Links

• is related to: HADOOP-16161 NetworkTopology#getWeightUsingNetworkLocation return unexpected result
• links to: GitHub Pull Request #4091