[HADOOP-16465] Tune S3AFileSystem.listLocatedStatus

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: fs/s3
• Assignee: Mukund Thakur
• Reporter: Steve Loughran
• Created: 29/Jul/19 15:32
• Updated: 30/Nov/20 10:32
• Resolved: 14/Apr/20 16:22

Description

Looking at logs of LocatedFileStatus/FileInputFormat scans; there's a needless call to getFileStatus whenever a S3AFileSystem.listLocatedStatus() call is made

1. S3AFileSystem.listLocatedStatus() does a getFileStatus call, returns the file status first
2. But if you look at all the uses in the MR code in FileInputFormat and LocatedFileStatusFetcher, they only call this method knowing the destination is a directory

Which means for every unguarded S3 path: two needless HEADS and a single entry LIST, before the real LIST is initiated.

If the S3A FS can assume that a dest is a non-empty directory, then it can go straight to the LIST operation, only falling back to the HEAD + HEAD +/ if that fails.

We could also think about doing the same for listStatus

Issue Links

• causes: HADOOP-17134 S3AFileSystem.listLocatedStatus(file) does a LIST even with S3Guard
• depends upon: HADOOP-16458 LocatedFileStatusFetcher scans failing intermittently against S3 store
• is blocked by: HADOOP-16697 audit/tune s3a authoritative flag in s3guard DDB Table
• links to: GitHub Pull Request #1943
• links to: GitHub Pull Request #1960