[HADOOP-18915] Tune/extend S3A http connection and thread pool settings

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0, 3.3.7-aws
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Steve Loughran
• Reporter: Ahmar Suhail
• Created: 29/Sep/23 10:32
• Updated: 25/Mar/24 23:08
• Resolved: 29/Nov/23 15:24

Description

Increases existing pool sizes, as with server scale and vector
IO, larger pools are needed

fs.s3a.connection.maximum 200
fs.s3a.threads.max 96

Adds new configuration options for v2 sdk internal timeouts,
both with default of 60s:

fs.s3a.connection.acquisition.timeout
fs.s3a.connection.idle.time

All the pool/timoeut options are covered in performance.md

Moves all timeout/duration options in the s3a FS to taking
temporal units (h, m, s, ms,...); retaining the previous default
unit (normally millisecond)

Adds a minimum duration for most of these, in order to recover from
deployments where a timeout has been set on the assumption the unit
was seconds, not millis.

Uses java.time.Duration throughout the codebase;
retaining the older numeric constants in
org.apache.hadoop.fs.s3a.Constants for backwards compatibility;
these are now deprecated.

Adds new class AWSApiCallTimeoutException to be raised on
sdk-related methods and also gateway timeouts. This is a subclass
of org.apache.hadoop.net.ConnectTimeoutException to support
existing retry logic.

+ reverted default value of fs.s3a.create.performance to false;
inadvertently set to true during testing.

Issue Links

• is related to: SPARK-47552 Set spark.hadoop.fs.s3a.connection.establish.timeout to 30s
• relates to: HADOOP-18889 S3A: V2 SDK client does not work with third-party store
• links to: AWS SDK issue #3066: S3ExpressIdentityCache does't pick up client timeout configurations
• links to: AWS SDK issue #4572
• links to: GitHub Pull Request #6180