[HADOOP-16202] Enhance openFile() for better read performance against object stores

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.4.0, 3.3.5
• Component/s: fs, fs/s3, tools/distcp
• Labels: pull-request-available
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 20/Mar/19 09:30
• Updated: 04/Dec/23 11:03
• Resolved: 24/Apr/22 16:40

Description

The openFile() builder API lets us add new options when reading a file

Add an option "fs.s3a.open.option.length" which takes a long and allows the length of the file to be declared. If set, no check for the existence of the file is issued when opening the file

Also: withFileStatus() to take any FileStatus implementation, rather than only S3AFileStatus -and not check that the path matches the path being opened. Needed to support viewFS-style wrapping and mounting.

and Adopt where appropriate to stop clusters with S3A reads switched to random IO from killing download/localization

• fs shell copyToLocal
• distcp
• IOUtils.copy

this patch MUST be followed by HADOOP-18410

Issue Links

• breaks: HADOOP-18410 S3AInputStream.unbuffer() async drain not releasing http connections
• breaks: MAPREDUCE-7391 TestLocalDistributedCacheManager failing after HADOOP-16202
• causes: HADOOP-18724 Open file fails with NumberFormatException for S3AFileSystem
• contains: HADOOP-17882 distcp to use openFile() with sequential IO; ranges of reads
• contains: HADOOP-17883 openFile() to add standard options for read policies, start/end ranges
• contains: HADOOP-17884 FsShell to use openFile(sequential) for reads
• incorporates: HADOOP-17896 ABFS: Stabilize openFile withStatus
• is depended upon by: HADOOP-18179 Boost S3A Stream Read Performance
• is depended upon by: YARN-10444 Node Manager to use openFile() with whole-file read policy for localizing files.
• is duplicated by: HADOOP-16387 S3A openFile() options to allow etag/version to be set
• is related to: HADOOP-15625 S3A input stream to use etags/version number to detect changed source files
• is related to: HADOOP-17682 ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
• is related to: HADOOP-16090 S3A Client to add explicit support for versioned stores
• is related to: HADOOP-18944 s3a openfile is using readahead rather than async drain threshold to calculate drain threshold
• is related to: HDFS-14478 Add libhdfs APIs for openFile
• relates to: HADOOP-17038 Support disabling buffered reads in ABFS positional reads
• relates to: HADOOP-16759 Filesystem openFile() builder to take a FileStatus param
• relates to: TEZ-4522 Use OpenFile where FileStatus is available.
• links to: GitHub Pull Request #2168
• links to: GitHub Pull Request #2584
• links to: GitHub Pull Request #4238
• links to: https://github.com/apache/hadoop/pull/2046

Sub-Tasks

• openFile() to add standard options for read policies, start/end ranges: Resolved
• FsShell to use openFile(sequential) for reads: Resolved