[HADOOP-16202] Stabilize openFile() and adopt internally

• Type: Sub-task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: fs, fs/s3, tools/distcp
• Labels: pull-request-available
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 20/Mar/19 09:30
• Updated: 1 hour ago

Description

The openFile() builder API lets us add new options when reading a file

Add an option "fs.s3a.open.option.length" which takes a long and allows the length of the file to be declared. If set, no check for the existence of the file is issued when opening the file

Also: withFileStatus() to take any FileStatus implementation, rather than only S3AFileStatus -and not check that the path matches the path being opened. Needed to support viewFS-style wrapping and mounting.

and Adopt where appropriate to stop clusters with S3A reads switched to random IO from killing download/localization

• fs shell copyToLocal
• distcp
• IOUtils.copy

Issue Links

• is depended upon by: YARN-10444 use openFile() with sequential IO for localizing files.
• is duplicated by: HADOOP-16387 S3A openFile() options to allow etag/version to be set
• is related to: HADOOP-15625 S3A input stream to use etags/version number to detect changed source files
• is related to: HADOOP-16090 S3A Client to add explicit support for versioned stores
• relates to: HADOOP-17038 Support disabling buffered reads in ABFS positional reads
• relates to: HADOOP-16759 Filesystem openFile() builder to take a FileStatus param
• links to: GitHub Pull Request #2168
• links to: GitHub Pull Request #2584
• links to: https://github.com/apache/hadoop/pull/2046