[HADOOP-18184] s3a prefetching stream to support unbuffer()

• Type: Sub-task
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: fs/s3
• Labels: pull-request-available
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 28/Mar/22 17:31
• Updated: 04/Jun/24 16:56

Description

Apache Impala uses unbuffer() to free up all client side resources held by a stream, so allowing it to have a map of available (path -> stream) objects, retained across queries.

This saves on having to reopen the files, with the cost of HEAD checks etc. S3AInputStream just closes its http connection. here there is a lot more state to discard, but all memory and file storage must be freed.

until this done, ITestS3AContractUnbuffer must skip when the prefetch stream is used.

its notable that the other tests don't fail, even though the stream doesn't implement the interface; the graceful degradation handles that. it should fail if the test xml resource says the stream does it, but that the stream capabilities say it doesn't.

Issue Links

• is depended upon by: HADOOP-19177 TestS3ACachingBlockManager fails intermittently in Yetus
• links to: GitHub Pull Request #5832
• links to: https://github.com/apache/hadoop/pull/4298