[HADOOP-17400] Optimize S3A for maximum performance in directory listings

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Component/s: fs/s3
• Target Version/s: 3.3.1
• Assignee: Mukund Thakur
• Reporter: Steve Loughran
• Created: 30/Nov/20 10:31
• Updated: 25/May/23 14:23
• Resolved: 17/May/21 05:39

Description

Make listing in applications as fast as we can get it especially for query planning.

• All operations used in listing directories for query planning etc to be optimized for their primary use: being passed directories (not files) and so make that faster even at the expense of more remote IO when handed files or empty directories.
• remove needless calls to S3 wherever possible (e.g. getFileStatus("/"), making bucket existence probes optional)
• Support/enable Asynchronous IO where possible.

Review higher level APIs (glob status) and uses on the FsShell and optimize their use by minimising invocations or FS API calls, with bonus goal of reduce/minimize risk of 404 caching.

Work with downstream projects to move to FS APIs which work best in this world -primarily the recursive listing operations and those which return RemoteIterator<FileStatus> -and so make any asynchronous page fetching operations useful.

Issue Links

• is related to: HADOOP-18753 S3AFileSystem doesn't consistently handle prefixes that are both files and directories between versions