-
Type:
Sub-task
-
Status: In Progress
-
Priority:
Major
-
Resolution: Unresolved
-
Affects Version/s: 3.3.0
-
Fix Version/s: None
-
Component/s: fs, fs/s3, tools/distcp
-
Labels:
The openFile() builder API lets us add new options when reading a file
Add an option "fs.s3a.open.option.length" which takes a long and allows the length of the file to be declared. If set, no check for the existence of the file is issued when opening the file
Also: withFileStatus() to take any FileStatus implementation, rather than only S3AFileStatus -and not check that the path matches the path being opened. Needed to support viewFS-style wrapping and mounting.
and Adopt where appropriate to stop clusters with S3A reads switched to random IO from killing download/localization
- fs shell copyToLocal
- distcp
- IOUtils.copy
- is depended upon by
-
YARN-10444 use openFile() with sequential IO for localizing files.
-
- In Progress
-
- is duplicated by
-
HADOOP-16387 S3A openFile() options to allow etag/version to be set
-
- Resolved
-
- is related to
-
HADOOP-15625 S3A input stream to use etags/version number to detect changed source files
-
- Resolved
-
-
HADOOP-16090 S3A Client to add explicit support for versioned stores
-
- Resolved
-
- relates to
-
HADOOP-17038 Support disabling buffered reads in ABFS positional reads
-
- Open
-
-
HADOOP-16759 Filesystem openFile() builder to take a FileStatus param
-
- Resolved
-
- links to