Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13693

Remove unnecessary search in INodeDirectory.addChild during image loading

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0, 3.1.4, 3.2.2
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added to their parent INode's map one by one. The adding procedure will search a position in the parent's map and then insert the child to the position. However, during image loading, the search is unnecessary since the insert position should always be at the end of the map given the sequence they are serialized on disk.

      Test this patch against a fsimage of a 70PB  cluster (200million files and 300million blocks), the image loading time be reduced from 1210 seconds to 1138 seconds.So it can reduce up to about 10% of time.

        Attachments

        1. HDFS-13693-001.patch
          2 kB
          zhouyingchao
        2. HDFS-13693-002.patch
          3 kB
          Lisheng Sun
        3. HDFS-13693-003.patch
          3 kB
          Lisheng Sun
        4. HDFS-13693-004.patch
          2 kB
          Lisheng Sun
        5. HDFS-13693-005.patch
          3 kB
          Lisheng Sun

          Issue Links

            Activity

              People

              • Assignee:
                leosun08 Lisheng Sun
                Reporter:
                sinago zhouyingchao
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: