Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15174

Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0, 3.1.4, 3.2.2
    • Component/s: None
    • Labels:
      None

      Description

      Calculating the size of each block and the size of the meta file requires io operation In ReplicaCachingGetSpaceUsed#refresh(). Pressure on disk performance when there are many block. HDFS-14313 is intended to reduce io operation. So get block size by ReplicaInfo and meta size by DataChecksum#getChecksumSize().

      @Override
        protected void refresh() {
            if (CollectionUtils.isNotEmpty(replicaInfos)) {
              for (ReplicaInfo replicaInfo : replicaInfos) {
                if (Objects.equals(replicaInfo.getVolume().getStorageID(),
                    volume.getStorageID())) {
                  dfsUsed += replicaInfo.getBlockDataLength();
                  dfsUsed += replicaInfo.getMetadataLength();
                  count++;
                }
              }
            }
        }
      

        Attachments

        1. HDFS-15174-001.patch
          1 kB
          Lisheng Sun

          Issue Links

            Activity

              People

              • Assignee:
                leosun08 Lisheng Sun
                Reporter:
                leosun08 Lisheng Sun
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: