[HADOOP-15547] WASB: improve listStatus performance

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.1, 3.0.2
• Fix Version/s: 2.10.0, 3.1.1
• Component/s: fs/azure
• Release Note: WASB: listStatus 10x performance improvement for listing 700,000 files
• Assignee: Thomas Marqardt
• Reporter: Thomas Marqardt
• Created: 18/Jun/18 04:07
• Updated: 04/Sep/18 15:33
• Resolved: 04/Sep/18 15:33

Description

The WASB implementation of Filesystem.listStatus is very slow due to O(n!) algorithm to remove duplicates and uses too much memory due to the extra conversion from BlobListItem to FileMetadata to FileStatus.  It takes over 30 minutes to list 700,000 files.