[HADOOP-16047] Avoid expensive rename when DistCp is writing to S3

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: fs/s3, tools/distcp
• Assignee: Andrew Olson
• Reporter: Andrew Olson
• Created: 14/Jan/19 15:28
• Updated: 25/Mar/20 19:11
• Resolved: 14/Jan/19 16:27

Description

When writing to an S3-based target, the temp file and rename logic in RetriableFileCopyCommand adds some unnecessary cost to the job, as the rename operation does a server-side copy + delete in S3 [1]. The renames are parallelized across all of the DistCp map tasks, so the severity is mitigated to some extent. However a configuration property to conditionally allow distributed copies to avoid that expense and write directly to the target path would improve performance considerably.

[1] https://github.com/apache/hadoop/blob/release-3.2.0-RC1/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md#object-stores-vs-filesystems

Issue Links

• duplicates: HADOOP-15281 Distcp to add no-rename copy option