[HADOOP-15281] Distcp to add no-rename copy option

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.3.0, 3.2.1, 3.1.3
• Component/s: tools/distcp
• Assignee: Andrew Olson
• Reporter: Steve Loughran
• Created: 02/Mar/18 18:03
• Updated: 14/Oct/19 15:37
• Resolved: 18/Feb/19 15:59

Description

Currently Distcp uploads a file by two strategies

1. append parts
2. copy to temp then rename

option 2 executes the following sequence in promoteTmpToTarget

    if ((fs.exists(target) && !fs.delete(target, false))
        || (!fs.exists(target.getParent()) && !fs.mkdirs(target.getParent()))
        || !fs.rename(tmpTarget, target)) {
      throw new IOException("Failed to promote tmp-file:" + tmpTarget
                              + " to: " + target);
    }

For any object store, that's a lot of HTTP requests; for S3A you are looking at 12+ requests and an O(data) copy call.

This is not a good upload strategy for any store which manifests its output atomically at the end of the write().

Proposed: add a switch to write directly to the dest path, which can be supplied as either a conf option (distcp.direct.write = true) or a CLI option (-direct).

Issue Links

• breaks: HADOOP-16096 HADOOP-15281/distcp -Xdirect needs to use commons-logging on 3.1
• is depended upon by: HADOOP-10007 distcp / mv is not working on ftp
• is depended upon by: HADOOP-15788 Improve Distcp for long-haul/cloud deployments
• is depended upon by: HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
• is duplicated by: HADOOP-15577 Update distcp to use zero-rename s3 committers
• is duplicated by: HADOOP-16047 Avoid expensive rename when DistCp is writing to S3
• is related to: HADOOP-13622 `-atomic` should not be supported while using `distcp` command in object file system
• is required by: HADOOP-10007 distcp / mv is not working on ftp
• relates to: HADOOP-15209 DistCp to eliminate needless deletion of files under already-deleted directories
• supercedes: HADOOP-16260 Allow Distcp to create a new tempTarget file per File
• supercedes: HADOOP-12046 Avoid creating "._COPYING_" temporary file when copying file to Swift file system
• links to: GitHub Pull Request #469