[HADOOP-18179] Boost S3A Stream Read Performance

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.2
• Component/s: fs/s3
• Assignee: Ahmar Suhail
• Reporter: Steve Loughran
• Created: 28/Mar/22 11:28
• Updated: 31/Jul/23 16:38

Description

calibrate S3A input stream performance against recent applications/data formats and improve where necessary.

HADOOP-18028 is a key part of this, but there are other issues/opertunities

1. we could add machine parsable trace-level logging in FSDataInputStream to collect stats on how stream apis are invoked, so collect data from real apps; analyze
2. implement those APIs which some apps use (ByteBufferPositionedReadable), not so much for direct implementation as to get better information from the app as its read plan
3. the `normal` mode doesn't switch from sequential on forward seeks. Is that always appropriate?
4. choose different buffering options when doing whole file IO vs sequential vs random

Issue Links

• depends upon: HADOOP-18028 High performance S3A input stream with prefetching & caching
• depends upon: HADOOP-16202 Enhance openFile() for better read performance against object stores
• is depended upon by: HADOOP-18477 Über-jira: S3A Hadoop 3.3.9 features
• relates to: HADOOP-17842 S3a parquet reads slow with Spark on Kubernetes (EKS)

Sub-Tasks

• s3a audit logs to publish range start/end of GET requests in audit header: Resolved