[HADOOP-17491] Implement listFiles(path, recursive=true)

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: fs/azure
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 25/Jan/21 10:33
• Updated: 25/Jan/21 10:36

Description

Azure storage and the abfs client offer recursive listings, but currently there's no listFiles() method to connect to it instead a recursive treewalk is used.

Implementing a listFiles(path, recursive) could offer significant speedups to applications which use it, for example HIVE-24669, and again, async prefetching could provide even more benefits

Would need changes in org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore to join everything up