[HADOOP-18103] High performance vectored read API in Hadoop

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.4
• Fix Version/s: 3.3.5
• Component/s: common, fs, fs/adl, fs/s3
• Labels: perfomance pull-request-available
• Assignee: Mukund Thakur
• Reporter: Mukund Thakur
• Created: 01/Feb/22 13:13
• Updated: 04/Oct/24 15:07
• Resolved: 09/Jan/23 18:47

Description

Add support for multiple ranged vectored read api in PositionedReadable. The default iterates through the ranges to read each synchronously, but the intent is that FSDataInputStream subclasses can make more efficient readers especially object stores implementation.

Issue Links

• causes: HADOOP-18854 add options to disable range merging of vectored io
• causes: HADOOP-18322 Yetus build failure in branch-3.3.
• is related to: ORC-1787 Failure on readDiskRangesVectored with concurrent record readers
• is related to: HADOOP-18287 Provide a shim library for modern FS APIs
• is related to: PARQUET-2171 Implement vectored IO in parquet file format
• is related to: HADOOP-18855 VectorIO API tuning/stabilization
• is related to: ORC-1570 Add `supportVectoredIO` API to `HadoopShimsCurrent` and use it
• relates to: HDDS-7298 Ozone support for VectorIO APIs in Hadoop
• links to: GitHub Pull Request #4476

Sub-Tasks

• Add a high-performance vectored read API.: Resolved
• Add configs to configure minSeekForVectorReads and maxReadSizeForVectorReads: Resolved
• Implement a variant of ElasticByteBufferPool which uses weak references for garbage collection.: Resolved
• Handle memory fragmentation in S3 Vectored IO implementation.: Resolved
• Vectored IO support for large S3 files.: Resolved
• Add input stream IOstats for vectored IO api in S3A.: Resolved
• Memory fragmentation in ChecksumFileSystem Vectored IO implementation.: Resolved
• Restrict vectoredIO threadpool to reduce memory pressure: Resolved
• Update previous index properly while validating overlapping ranges.: Resolved
• Propagate vectored s3a input stream stats to file system stats.: Resolved
• Improve vectored IO api spec.: Resolved
• Improve VectoredReadUtils#readVectored() for direct buffers: Resolved
• Fix VectoredIO for LocalFileSystem when checksum is enabled.: Resolved
• Vectored IO: Threadpool should be closed on interrupts or during close calls: Resolved
• ITestS3AContractVectoredRead.testStopVectoredIoOperationsUnbuffer failing: Resolved
• Add an integration test to process data asynchronously during vectored read.: Resolved
• VectorIO FileRange type to support a "reference" field: Resolved
• ChecksumFileSystem::readVectored might return byte buffers not positioned at 0: Resolved