[HDFS-16059] dfsadmin -listOpenFiles -blockingDecommission can miss some files

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: dfsadmin
• Assignee: Unassigned
• Reporter: Akira Ajisaka
• Created: 09/Jun/21 08:40
• Updated: 10/Nov/21 06:51

Description

While reviewing HDFS-13671, I found "dfsadmin -listOpenFiles -blockingDecommission" can drop some files.

https://github.com/apache/hadoop/pull/3065#discussion_r647396463

If the DataNodes have the following open files and we want to list all the open files:

DN1: [1001, 1002, 1003, ... , 2000]
DN2: [1, 2, 3, ... , 1000]

At first getFilesBlockingDecom(0, "/") is called and it returns [1001, 1002, ... , 2000] because it reached max size (=1000), and next getFilesBlockingDecom(2000, "/") is called because the last inode Id of the previous result is 2000. That way the open files of DN2 is missed

Issue Links

• relates to: HDFS-13671 Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet
• relates to: HDFS-11847 Enhance dfsadmin listOpenFiles command to list files blocking datanode decommissioning