[HADOOP-16313] multipart/huge file upload tests to look at checksums returned

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: fs/s3, test
• Target Version/s: 3.5.0
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 15/May/19 09:48
• Updated: 03/Dec/24 13:39

Description

Apparently some third party S3-API object stores have empty etags for multipart uploads.

We don't have any tests to explore this (just small file uploads).

Proposed

*set fs.s3a.etag.checksum.enabled" before huge file, MPU and committer tests

• warn (fail?) if nothing comes back.

If true, it means that the store isn't going to be detect changes to a large file during copy/read

Issue Links

• relates to: HADOOP-15625 S3A input stream to use etags/version number to detect changed source files
• relates to: HADOOP-16085 S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite