[HADOOP-18347] Restrict vectoredIO threadpool to reduce memory pressure

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.5
• Component/s: common, fs, fs/adl, fs/s3
• Labels: performance pull-request-available
• Assignee: Mukund Thakur
• Reporter: Rajesh Balamohan
• Created: 18/Jul/22 22:09
• Updated: 07/Sep/23 16:29
• Resolved: 28/Sep/22 17:48

Description

https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L964-L967

Currently, it fetches all the ranges with unbounded threadpool. This will not cause memory pressures with standard benchmarks like TPCDS. However, when large number of ranges are present with large files, this could potentially spike up memory usage of the task. Limiting the threadpool size could reduce the memory usage.

Issue Links

• fixes: HADOOP-18447 Vectored IO: Threadpool should be closed on interrupts or during close calls
• links to: GitHub Pull Request #4918