[HDFS-324] Generate a network infrastructre map

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: Unassigned
• Reporter: Kai Mosebach
• Created: 05/Sep/08 15:18
• Updated: 19/Jul/14 18:57
• Resolved: 19/Jul/14 18:57

Description

Assuming an inhomogeneous network it might be sensible to not only collect metrics about to and from the nodes but also

• collect latency between nodes
• network io between nodes
• daytime depending network io between nodes (i.e. network is much slower in the morning time and in the early evening time)
• network failures

Assuming the we collect aging information over a period of time, it would allow us to create a "learning cloud" in senses of its infrastructure. From this a network map we can

• see subclusters of high speed linked nodes*
• see unreliable connections between nodes
• see heavily used links over the time

These information can be refed into the DFS (for data distribution as well as for the balancer) logic to increase its reliability and its performance a lot.

*Note : This could already be managed with the rack awareness, but the aging approach would make this much more fine grained and in an automatic manner.

Issue Links

• is duplicated by: HDFS-289 HDFS should blacklist datanodes that are not performing well
• is related to: HADOOP-2830 Need to instrument Hadoop to get comprehensive network traffic metrics