[HADOOP-19317] S3A: fs.s3a.connection.expect.continue controls 100 CONTINUE behavior

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.1
• Fix Version/s: 3.3.9, 3.5.0, 3.4.2
• Component/s: fs/s3
• Labels: pull-request-available
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 24/Oct/24 13:58
• Updated: 25/Nov/24 19:03
• Resolved: 19/Nov/24 14:40

Description

New option

  fs.s3a.connection.expect.continue

This controls whether or not an PUT request to the S3 store
sets the "Expect: 100-continue" header and awaits a 100 CONTINUE
response before uploading any data.

This allows for throttling and other problems to be detected fast.

The default is "true" -the header is sent.

It seems like either the AWS SDK or http client libraries underneath don't recognise a dead HTTPS connection when using this.

Leaving as enabled as it can cope with heavy load better (clients can get a 503 and know to back off before uploading any data.

Issue Links

• links to: GitHub Pull Request #7134
• links to: GitHub Pull Request #7178