Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14419

Avoid repeated calls to the listOpenFiles function

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: 3.3.0
    • Component/s: namenode, performance
    • Labels:
      None
    • Target Version/s:

      Description

      `hdfs dfsadmin -listOpenFiles -path /any/path` will request all opened files. In the NameNode side, the function LeaseManager.java#getUnderConstructionFiles will be called.

      When there are only N(<maxListOpenFilesResponses, default is 100) files meet the conditions, but the leaseManager contains M(>maxListOpenFilesResponses) files, we will scan all leases. Finally, the hasMore will be set true and the openFileEntries contains N files, the scenario will cause listOpenFiles() be called again.

      If M is greater more than N, the two calls to getUnderConstructionFiles will impact the NameNode performance.

        Attachments

        1. HDFS-14419.001.patch
          1 kB
          HuangTao
        2. HDFS-14419.002.patch
          1 kB
          HuangTao

          Issue Links

            Activity

              People

              • Assignee:
                marvelrock HuangTao
                Reporter:
                marvelrock HuangTao
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: