[HDFS-14313] Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0, 2.7.0, 2.8.0, 2.9.0, 3.0.0, 3.1.0
• Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 3.1.4, 3.2.2
• Component/s: datanode, performance
• Hadoop Flags: Reviewed
• Assignee: Lisheng Sun
• Reporter: Lisheng Sun
• Created: 23/Feb/19 05:48
• Updated: 20/Feb/20 23:56
• Resolved: 07/Aug/19 15:22

Description

There are two ways of DU/DF getting used space that are insufficient.

1.  Running DU across lots of disks is very expensive and running all of the processes at the same time creates a noticeable IO spike.
2.  Running DF is inaccurate when the disk sharing by multiple datanode or other servers.

 Getting hdfs used space from  FsDatasetImpl#volumeMap#ReplicaInfos in memory is very small and accurate. 

Issue Links

• causes: HDFS-14986 ReplicaCachingGetSpaceUsed throws ConcurrentModificationException
• is related to: HDFS-15174 Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations
• is related to: HDFS-15039 Cache meta file length of FinalizedReplica to reduce call File.length()
• relates to: HADOOP-12973 make DU pluggable
• relates to: HADOOP-12974 Create a CachingGetSpaceUsed implementation that uses df
• relates to: HADOOP-9884 Hadoop calling du -sk is expensive
• links to: GitHub Pull Request #1011