[HADOOP-9884] Hadoop calling du -sk is expensive

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Assignee: Unassigned
• Reporter: Alex Newman
• Created: 19/Aug/13 19:12
• Updated: 23/Feb/19 05:49
• Resolved: 08/May/17 08:19

Description

On numerous occasions we've had customers worry about slowness while hadoop calls du -sk underneath the hood. For most of these users getting the information from df would be sufficient and much faster. In fact there is a hack going around, that is quiet common that replaces df with du. Sometimes people have to tune the vcache. What if we just allowed users to use the df information instead of the du information with a patch and config setting. I'd be glad to code it up

Issue Links

• is related to: HDFS-14313 Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du

Sub-Tasks

• make DU pluggable: Resolved
• Create a CachingGetSpaceUsed implementation that uses df: Resolved
• Add jitter to CachingGetSpaceUsed's thread: Resolved
• DF implementation of CachingGetSpaceUsed makes DFS Used size not correct: Resolved