[HADOOP-19033] S3A: disable checksums when fs.s3a.checksum.validation = false

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0, 3.5.0
• Component/s: fs/s3
• Labels: pull-request-available
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 10/Jan/24 17:18
• Updated: 06/Mar/24 11:49
• Resolved: 19/Jan/24 14:09

Description

AWS v2 sdk turns on client-side checksum validation; this kills performance

Given we are using TLS to download from AWS s3, there's implicit channel checksumming going on on, that's along with the IPv4 TCP checksumming.

We don't need it, all it does is slow us down.

proposed: disable in DefaultS3ClientFactory

I don't want to add an option to enable it as it only complicates life (yet another config option), but I am open to persuasion

Issue Links

• links to: GitHub Pull Request #6441