[HDFS-14419] Avoid repeated calls to the listOpenFiles function

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: namenode, performance
• Target Version/s: 3.3.0
• Assignee: HuangTao
• Reporter: HuangTao
• Created: 09/Apr/19 10:18
• Updated: 31/Jul/19 01:40
• Resolved: 30/Jul/19 23:51

Description

`hdfs dfsadmin -listOpenFiles -path /any/path` will request all opened files. In the NameNode side, the function LeaseManager.java#getUnderConstructionFiles will be called.

When there are only N(<maxListOpenFilesResponses, default is 100) files meet the conditions, but the leaseManager contains M(>maxListOpenFilesResponses) files, we will scan all leases. Finally, the hasMore will be set true and the openFileEntries contains N files, the scenario will cause listOpenFiles() be called again.

If M is greater more than N, the two calls to getUnderConstructionFiles will impact the NameNode performance.

Issue Links

• relates to: HDFS-10480 Add an admin command to list currently open files