[AVRO-3594] FsInput to use openFile() API for cloud storage read performance

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.11.2
• Component/s: java
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 04/Aug/22 15:39
• Updated: 12/Aug/22 13:48

Description

avro can now use the FileSystem.openFile() API to open a file on a hadoop filesystem connector (HADOOP-15229).

by setting the file length and fadvise policy through opt() calls, the clients can

• skip a HEAD request when opening a file
• optimise the ranges of GET request for sequential access, even in clusters where s3a has been configured to use random iO (which some hive clusters do)

filesystems/releases which don't recognise the options added in HADOOP-16202 will ignore them; the api will fall back to classic open(path) API call if the connector doesn't have a custom implementation.

Issue Links

• links to: GitHub Pull Request #1807