[SPARK-24280] Speed up indexing of files in object stores by using listFiles(path, recursive=true)

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 15/May/18 10:49
• Updated: 08/Oct/19 05:43
• Resolved: 08/Oct/19 05:43

Description

Object stores which mock directory structures can have awful performance when a treewalk is executed on them. The FileSystem.listFiles(path, recursive=true) call can reduce this overhead to ~ O(1) by doing bulk paged listing of all files under a path. For filesystems without this operation, the treewalk is implemented behind the scenes: no cost compared to today's code, just a simplification

Issue Links

• relates to: SPARK-17593 list files on s3 very slow
• relates to: SPARK-20255 FileIndex hierarchy inconsistency