[MAPREDUCE-5899] Support incremental data copy in DistCp

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.5.0
• Component/s: distcp
• Hadoop Flags: Reviewed
• Assignee: Jing Zhao
• Reporter: Jing Zhao
• Created: 14/May/14 02:32
• Updated: 06/Mar/18 02:27
• Resolved: 22/May/14 18:50

Description

Currently when doing distcp with -update option, for two files with the same file names but with different file length or checksum, we overwrite the whole file. It will be good if we can detect the case where (sourceFile = targetFile + appended_data), and only transfer the appended data segment to the target. This will be very useful if we're doing incremental distcp.

Issue Links

• is related to: HADOOP-15292 Distcp's use of pread is slowing it down.
• relates to: MAPREDUCE-6471 Document distcp incremental copy