[HADOOP-15273] distcp can't handle remote stores with different checksum algorithms

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0, 3.0.3
• Component/s: tools/distcp
• Target Version/s: 3.1.0
• Assignee: Stephen O'Donnell
• Reporter: Steve Loughran
• Created: 28/Feb/18 15:21
• Updated: 20/Feb/20 12:01
• Resolved: 08/Mar/18 11:28

Description

When using distcp without -skipcrcchecks . If there's a checksum mismatch between src and dest store types (e.g hdfs to s3), then the error message will talk about blocksize, even when its the underlying checksum protocol itself which is the cause for failure

Source and target differ in block-size. Use -pb to preserve block-sizes during copy. Alternatively, skip checksum-checks altogether, using -skipCrc. (NOTE: By skipping checksums, one runs the risk of masking data-corruption during file-transfer.)

update: the CRC check takes always place on a distcp upload before the file is renamed into place. and you can't disable it then

Issue Links

• depends upon: HADOOP-15297 Make S3A etag => checksum feature optional
• is depended upon by: HADOOP-14831 Über-jira: S3a phase IV: Hadoop 3.1 features
• is related to: HADOOP-16536 Backport HADOOP-16158 and HADOOP-15273 to branch-2
• relates to: HADOOP-13282 S3 blob etags to be made visible in S3A status/getFileChecksum() calls