[HDFS-15174] Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0, 3.1.4, 3.2.2
• Assignee: Lisheng Sun
• Reporter: Lisheng Sun
• Created: 17/Feb/20 07:27
• Updated: 24/Feb/20 21:54
• Resolved: 24/Feb/20 21:40

Description

Calculating the size of each block and the size of the meta file requires io operation In ReplicaCachingGetSpaceUsed#refresh(). Pressure on disk performance when there are many block. HDFS-14313 is intended to reduce io operation. So get block size by ReplicaInfo and meta size by DataChecksum#getChecksumSize().

@Override
  protected void refresh() {
      if (CollectionUtils.isNotEmpty(replicaInfos)) {
        for (ReplicaInfo replicaInfo : replicaInfos) {
          if (Objects.equals(replicaInfo.getVolume().getStorageID(),
              volume.getStorageID())) {
            dfsUsed += replicaInfo.getBlockDataLength();
            dfsUsed += replicaInfo.getMetadataLength();
            count++;
          }
        }
      }
  }

Issue Links

• relates to: HDFS-14313 Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du