[HDFS-6840] Clients are always sent to the same datanode when read is off rack

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.5.0
• Fix Version/s: 2.6.0
• Target Version/s: 2.6.0
• Assignee: Andrew Wang
• Reporter: Jason Darrell Lowe
• Created: 12/Aug/14 00:46
• Updated: 12/Apr/17 12:41
• Resolved: 19/Sep/14 01:14

Description

After HDFS-6268 the sorting order of block locations is deterministic for a given block and locality level (e.g.: local, rack. off-rack), so off-rack clients all see the same datanode for the same block. This leads to very poor behavior in distributed cache localization and other scenarios where many clients all want the same block data at approximately the same time. The one datanode is crushed by the load while the other replicas only handle local and rack-local requests.

Issue Links

• duplicates: HDFS-4253 block replica reads get hot-spots due to NetworkTopology#pseudoSortByDistance
• is broken by: HDFS-6268 Better sorting in NetworkTopology#pseudoSortByDistance when no local node is found
• relates to: HADOOP-11107 Improve tests to not rely on JDK random implementation
• relates to: HDFS-6701 Make seed optional in NetworkTopology#sortByDistance