[HDFS-15039] Cache meta file length of FinalizedReplica to reduce call File.length()

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.3.0
• Component/s: datanode
• Assignee: Yang Yun
• Reporter: Yang Yun
• Created: 09/Dec/19 03:31
• Updated: 20/Apr/21 08:22
• Resolved: 13/Mar/20 09:56

Description

When use ReplicaCachingGetSpaceUsed to get the volume space used.  It will call File.length() for every meta file of replica. That add more disk IO, we found the slow log as below. For finalized replica, the size of meta file is not changed, i think we can cache the value.

org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed: Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas size: 1166 dfsUsed: 72227113183 on volume: DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms

Issue Links

• relates to: HDFS-14313 Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du