[HADOOP-18410] S3AInputStream.unbuffer() async drain not releasing http connections

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.3.5
• Fix Version/s: 3.4.0, 3.3.5
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.5
• Hadoop Flags: Reviewed
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 19/Aug/22 10:00
• Updated: 28/Jan/24 00:12
• Resolved: 05/Sep/22 13:23

Description

Impala tcp-ds setup to s3 is hitting problems with timeout fetching http connections from the s3a fs pool. Disabling s3a async drain makes this problem go away. assumption, either those async ops are blocking, or they are not releasing references properly.

Issue Links

• is broken by: HADOOP-16202 Enhance openFile() for better read performance against object stores
• links to: GitHub Pull Request #4766
• links to: GitHub Pull Request #4839