[HADOOP-18287] Provide a shim library for modern FS APIs

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: fs
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 10/Jun/22 16:48
• Updated: 13/Jun/22 10:21

Description

Add a shim library to give libraries and applications built against hadoop 3.2 access to APIs and features in later versions, especially those delivering higher performance in cloud deployments. This will give them the ability to invoke those APIs when available, so gain from the work everyone has done. Key APIs are:

• openFile
• ByteBufferPositionedReadable
• Vectored IO

The library will either downgrade gracefully to existing code (openFile) or simply thrown UnsupportedException when invoked -but offer probes for every operation before invocation.

This module will compile against hadoop 3.2.0; it will be tested against that and later releases.

We can and should release this on a different schedule; though ideally we should issue releases in sync with new hadoop releases adding new supported API calls.

For that reason I think we could consider having separate git repository for it. Verifying that the shim works against hadoop PRs could actually become one of our regression tests -indeed, it should become one.

Issue Links

• is related to: PARQUET-2158 Upgrade Hadoop dependency to version 3.2.0
• relates to: HADOOP-15229 Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.
• relates to: HADOOP-15691 Add PathCapabilities to FS and FC to complement StreamCapabilities
• relates to: HADOOP-18103 High performance vectored read API in Hadoop
• relates to: HDFS-3246 pRead equivalent for direct read path
• links to: github/fs-api-shim