[HADOOP-16536] Backport HADOOP-16158 and HADOOP-15273 to branch-2

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.9.2
• Component/s: tools/distcp
• Assignee: Kai Xie
• Reporter: Kai Xie
• Created: 28/Aug/19 14:26
• Updated: 09/Dec/19 19:40

Description

Copying blocks in parallel (enabled when blocks per chunk > 0) is a great DistCp improvement that can hugely speed up copying big files. 

But its checksum validation is skipped, e.g. in `RetriableFileCopyCommand.java`

 

if (!source.isSplit()) {
  compareCheckSums(sourceFS, source.getPath(), sourceChecksum,
      targetFS, targetPath);
}

and this could result in checksum/data mismatch without notifying developers/users (e.g. HADOOP-16049).

I'd like to provide a patch to add the checksum validation.

 

Issue Links

• is a clone of: HADOOP-16158 DistCp to support checksum validation when copy blocks in parallel
• relates to: HADOOP-15273 distcp can't handle remote stores with different checksum algorithms
• relates to: HADOOP-16158 DistCp to support checksum validation when copy blocks in parallel
• links to: GitHub Pull Request #1368