[HADOOP-15229] Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: fs, fs/azure, fs/s3
• Target Version/s: 3.3.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 13/Feb/18 22:12
• Updated: 10/Jun/22 16:50
• Resolved: 05/Feb/19 12:06

Description

Replicate HDFS-1170 and HADOOP-14365 with an API to open files.

A key requirement of this is not HDFS, it's to put in the fadvise policy for working with object stores, where getting the decision to do a full GET and TCP abort on seek vs smaller GETs is fundamentally different: the wrong option can cost you minutes. S3A and Azure both have adaptive policies now (first backward seek), but they still don't do it that well.

Columnar formats (ORC, Parquet) should be able to say "fs.input.fadvise" "random" as an option when they open files; I can imagine other options too.

The Builder model of eddyxu is the one to mimic, method for method. Ideally with as much code reuse as possible

Issue Links

• causes: HADOOP-16480 S3 Select Exceptions are not being converted to IOEs
• causes: MAPREDUCE-7184 TestJobCounters byte counters omitting crc file bytes read
• causes: HADOOP-16106 hadoop-aws project javadoc does not compile
• incorporates: HADOOP-15364 Add support for S3 Select to S3A
• is depended upon by: HADOOP-15963 Add ABFS support for Async Scatter/Gather IO
• is depended upon by: HADOOP-15364 Add support for S3 Select to S3A
• is depended upon by: HADOOP-15964 Add S3A support for Async Scatter/Gather IO
• is depended upon by: MAPREDUCE-7182 MapReduce input format/record readers to support S3 select queries
• is related to: HADOOP-11867 Add a high-performance vectored read API.
• is related to: HADOOP-15949 open(PathHandle) spec and implementations ambiguous about what to raise when path is deleted
• is related to: HDFS-14111 hdfsOpenFile on HDFS causes unnecessary IO from file offset 0
• is related to: HADOOP-18287 Provide a shim library for modern FS APIs
• is required by: HADOOP-15625 S3A input stream to use etags/version number to detect changed source files
• relates to: HADOOP-13327 Add OutputStream + Syncable to the Filesystem Specification
• relates to: HADOOP-14365 Stabilise FileSystem builder-based create API
• relates to: HADOOP-15691 Add PathCapabilities to FS and FC to complement StreamCapabilities
• relates to: HDFS-2744 Extend FSDataInputStream to allow fadvise
• relates to: HDFS-11170 Add builder-based create API to FileSystem
• relates to: HDFS-14478 Add libhdfs APIs for openFile