[HADOOP-15224] builld up md5 checksum as blocks are built in S3ABlockOutputStream; validate upload

• Type: Sub-task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: fs/s3
• Target Version/s: 3.5.0
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 13/Feb/18 12:13
• Updated: 16/Feb/24 12:45

Description

rdblue reports sometimes he sees corrupt data on S3. Given MD5 checks from upload to S3, its likelier to have happened in VM RAM, HDD or nearby.

If the MD5 checksum for each block was built up as data was written to it, and checked against the etag RAM/HDD storage of the saved blocks could be removed as sources of corruption

The obvious place would be org.apache.hadoop.fs.s3a.S3ADataBlocks.DataBlock

Issue Links

• is depended upon by: HADOOP-19080 S3A to support writing to object lock buckets
• relates to: BEAM-5196 Add MD5 consistency check on S3 uploads (writes)