[HDFS-97] DFS should detect slow links(nodes) and avoid them

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: Unassigned
• Reporter: Runping Qi
• Created: 18/Jun/08 01:20
• Updated: 21/Jul/14 20:03
• Resolved: 19/Jul/14 18:54

Description

The current DFS does not detect slow links (nodes).
Thus, when a node or its network link is slow, it may affect the overall system performance significantly.
Specifically, when a map job needs to read data from such a node, it may progress 10X slower.
And when a DFS data node pipeline consists of such a node, the write performance degrades significantly.
This may lead to some long tails for map/reduce jobs. We have experienced such behaviors quite often.

Issue Links

• is related to: HDFS-289 HDFS should blacklist datanodes that are not performing well
• is related to: HDFS-6450 Support non-positional hedged reads in HDFS