[HADOOP-13637] improve setting of max connections in AWS client

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 2.8.0
• Fix Version/s: 3.3.2
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 21/Sep/16 19:52
• Updated: 17/Sep/21 14:07
• Resolved: 17/Sep/21 14:07

Description

things can go badly wrong if the S3A FS creates a thread pool for IO > than the number of pooled AWS http connections (set by property MAXIMUM_CONNECTIONS); you also need some for any other IO requests coming in.

The max connections property is currently independent of thread pool size, and has a default value of 1.

this is why there is a troubleshooting section in the docs showing the stack trace and instructions to fix".

Better: have a dynamic minimum like thread pool size + n, for a value of n to be chosen.

Issue Links

• duplicates: HADOOP-17869 fs.s3a.connection.maximum should be bigger than fs.s3a.threads.max
• is related to: HADOOP-16618 increase the default number of threads and http connections in S3A