[HADOOP-16759] Filesystem openFile() builder to take a FileStatus param

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: fs, fs/azure, fs/s3
• Target Version/s: 3.3.0
• Hadoop Flags: Reviewed
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 12/Dec/19 12:49
• Updated: 04/Dec/23 11:03
• Resolved: 21/Jan/20 22:36

Description

Let us pass in a file status to openFile() so that S3A & ABFS will skip their own HEAD requests just to see if a file is there, a normal file and get its length + etag,

    CompletableFuture<FSDataInputStream> streamF = fs.openFile(stat.getPath())
      .withFileStatus(stat).build()

code opening files off a listing of everything in a directory can eliminate a lot of requests here.

Also: change the specification of openFile's completable future to say "returned stream may only raise FNFE, access restrictions on the first read"

That is: it's not just potentially an async open, it's possibly lazy evaluated entirely.

Issue Links

• blocks: HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
• breaks: HADOOP-16853 ITestS3GuardOutOfBandOperations failing on versioned S3 buckets
• causes: HADOOP-16827 TestHarFileSystem.testInheritedMethodsImplemented broken
• is depended upon by: HDFS-14478 Add libhdfs APIs for openFile
• is related to: HADOOP-16101 Use lighter-weight alternatives to innerGetFileStatus where possible
• is related to: HADOOP-16202 Enhance openFile() for better read performance against object stores
• relates to: TEZ-4522 Use OpenFile where FileStatus is available.
• links to: GitHub Pull Request #1761
• links to: GitHub Pull Request #1850