[HDFS-3751] DN should log warnings for lengthy disk IOs

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.2.0, 2.0.0-alpha
• Component/s: datanode
• Target Version/s: 2.0.2-alpha, 1.3.0
• Assignee: Colin McCabe
• Reporter: Todd Lipcon
• Created: 01/Aug/12 20:47
• Updated: 17/Mar/14 17:19

Description

Occasionally failing disks or other OS-and-below issues cause a single IO to take tens of seconds, or even minutes in the case of failures. This often results in timeout exceptions at the client side which are hard to diagnose. It would be easier to root-cause these issues if the DN logged a WARN like "IO of 64kb to volume /data/1/dfs/dn for block 12345234 client 1.2.3.4 took 61.3 seconds" or somesuch.

Issue Links

• is related to: HDFS-6110 adding more slow action log in critical write path