Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.3.0
Description
The openFile() builder API lets us add new options when reading a file
Add an option "fs.s3a.open.option.length" which takes a long and allows the length of the file to be declared. If set, no check for the existence of the file is issued when opening the file
Also: withFileStatus() to take any FileStatus implementation, rather than only S3AFileStatus -and not check that the path matches the path being opened. Needed to support viewFS-style wrapping and mounting.
and Adopt where appropriate to stop clusters with S3A reads switched to random IO from killing download/localization
- fs shell copyToLocal
- distcp
- IOUtils.copy
this patch MUST be followed by HADOOP-18410
Attachments
Issue Links
- breaks
-
HADOOP-18410 S3AInputStream.unbuffer() async drain not releasing http connections
-
- Resolved
-
-
MAPREDUCE-7391 TestLocalDistributedCacheManager failing after HADOOP-16202
-
- Resolved
-
- causes
-
HADOOP-18724 Open file fails with NumberFormatException for S3AFileSystem
-
- Resolved
-
- contains
-
HADOOP-17882 distcp to use openFile() with sequential IO; ranges of reads
-
- Resolved
-
-
HADOOP-17883 openFile() to add standard options for read policies, start/end ranges
-
- Resolved
-
-
HADOOP-17884 FsShell to use openFile(sequential) for reads
-
- Resolved
-
- incorporates
-
HADOOP-17896 ABFS: Stabilize openFile withStatus
-
- Resolved
-
- is depended upon by
-
HADOOP-18179 Boost S3A Stream Read Performance
-
- Open
-
-
YARN-10444 Node Manager to use openFile() with whole-file read policy for localizing files.
-
- Resolved
-
- is duplicated by
-
HADOOP-16387 S3A openFile() options to allow etag/version to be set
-
- Reopened
-
- is related to
-
HADOOP-15625 S3A input stream to use etags/version number to detect changed source files
-
- Resolved
-
-
HADOOP-17682 ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
-
- Resolved
-
-
HADOOP-16090 S3A Client to add explicit support for versioned stores
-
- Resolved
-
-
HADOOP-18944 s3a openfile is using readahead rather than async drain threshold to calculate drain threshold
-
- Open
-
-
HDFS-14478 Add libhdfs APIs for openFile
-
- Resolved
-
- relates to
-
HADOOP-17038 Support disabling buffered reads in ABFS positional reads
-
- Resolved
-
-
HADOOP-16759 Filesystem openFile() builder to take a FileStatus param
-
- Resolved
-
-
TEZ-4522 Use OpenFile where FileStatus is available.
-
- Resolved
-
- links to