[HADOOP-13371] S3A globber to use bulk listObject call over recursive directory scan

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.8.0
• Component/s: fs, fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 13/Jul/16 14:57
• Updated: 26/Jul/19 20:21
• Resolved: 12/Feb/18 13:11

Description

HADOOP-13208 produces O(1) listing of directory trees in FileSystem.listStatus calls, but doesn't do anything for FileSystem.globStatus(), which uses a completely different codepath, one which does a selective recursive scan by pattern matching as it goes down, filtering out those patterns which don't match. Cost is O(matching-directories) + cost of examining the files.

It should be possible to do the glob status listing in S3A not through the filtered treewalk, but through a list + filter operation. This would be an O(files) lookup before any filtering took place.

Issue Links

• depends upon: HADOOP-13208 S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories
• is related to: HADOOP-14235 S3A Path does not understand colon (:) when globbing
• is superceded by: HADOOP-16458 LocatedFileStatusFetcher scans failing intermittently against S3 store
• links to: GitHub Pull Request #203
• links to: GitHub Pull Request #204