[HADOOP-16756] distcp -update to S3A; abfs, etc always overwrites due to block size mismatch

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.0.4, 3.2.2, 3.3.1
• Component/s: fs/s3, tools/distcp
• Assignee: Steve Loughran
• Reporter: Daisuke Kobayashi
• Created: 10/Dec/19 10:06
• Updated: 10/Jun/21 07:56
• Resolved: 15/May/20 13:09

Description

Distcp over S3A always copies all source files no matter the files are changed or not. This is opposite to the statement in the doc below.

http://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html

And to use -update to only copy changed files.

CopyMapper compares file length as well as block size before copying. While the file length should match, the block size does not. This is apparently because the returned block size from S3A is always 32MB.

https://github.com/apache/hadoop/blob/release-3.2.0-RC1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java#L348

I'd suppose we should update the documentation or make code change.

Issue Links

• duplicates: HADOOP-17256 DistCp -update option will be invalid when distcp files from hdfs to S3
• is broken by: HADOOP-8143 Change distcp to have -pb on by default
• is duplicated by: HADOOP-15300 distcp -update to WASB and ADL copies up all the files, always
• is related to: HADOOP-16932 distcp copy calls getFileStatus() needlessly and can fail against S3