[HADOOP-13282] S3 blob etags to be made visible in S3A status/getFileChecksum() calls

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.9.0
• Fix Version/s: 3.1.0
• Component/s: fs/s3
• Target Version/s: 3.1.0
• Hadoop Flags: Incompatible change

Release Note

now that S3A has a checksum, you need to explicitly disable checksums when uploading from HDFS : use -skipCrc

checksum verification does work between s3a buckets, provided the block size on uploads was identical

• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 16/Jun/16 20:01
• Updated: 07/Mar/18 20:57
• Resolved: 21/Dec/17 15:04

Description

If the etags of blobs were exported via getFileChecksum(), it'd be possible to probe for a blob being in sync with a local file. Distcp could use this to decide whether to skip a file or not.

Now, there's a problem there: distcp needs source and dest filesystems to implement the same algorithm. It'd only work out the box if you were copying between S3 instances. There are also quirks with encryption and multipart: s3 docs. At the very least, it's something which could be used when indexing the FS, to check for changes later.

Issue Links

• causes: HADOOP-15297 Make S3A etag => checksum feature optional
• is depended upon by: IMPALA-6057 Cache Remote Reads
• is related to: HADOOP-15273 distcp can't handle remote stores with different checksum algorithms