[HADOOP-19097] core-default fs.s3a.connection.establish.timeout value too low -warning always printed

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0, 3.4.1
• Component/s: fs/s3
• Labels: pull-request-available
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 29/Feb/24 10:43
• Updated: 25/Mar/24 23:08
• Resolved: 05/Mar/24 10:20

Description

caused by HADOOP-18915.

in core-default we set the value of fs.s3a.connection.establish.timeout to 5s

<property>
  <name>fs.s3a.connection.establish.timeout</name>
  <value>5s</value>
</property>

but there is a minimum of 15s, so this prints a warning

2024-02-29 10:39:27,369 WARN impl.ConfigurationHelper: Option fs.s3a.connection.establish.timeout is too low (5,000 ms). Setting to 15,000 ms instead

Issue Links

• is related to: SPARK-47552 Set spark.hadoop.fs.s3a.connection.establish.timeout to 30s
• links to: GitHub Pull Request #6601