[HADOOP-16158] DistCp to support checksum validation when copy blocks in parallel

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0, 2.9.2, 3.0.3, 3.1.2
• Fix Version/s: 3.3.0, 3.2.1, 3.1.3
• Component/s: tools/distcp
• Assignee: Kai Xie Assign to me
• Reporter: Kai Xie
• Created: 03/Mar/19 07:47
• Updated: 02/Oct/19 17:12
• Resolved: 19/Aug/19 01:57

Description

Copying blocks in parallel (enabled when blocks per chunk > 0) is a great DistCp improvement that can hugely speed up copying big files. 

But its checksum validation is skipped, e.g. in `RetriableFileCopyCommand.java`

 

if (!source.isSplit()) {
  compareCheckSums(sourceFS, source.getPath(), sourceChecksum,
      targetFS, targetPath);
}

and this could result in checksum/data mismatch without notifying developers/users (e.g. HADOOP-16049).

I'd like to provide a patch to add the checksum validation.

 

Issue Links

• is cloned by: HADOOP-16536 Backport HADOOP-16158 and HADOOP-15273 to branch-2
• is related to: HADOOP-16049 DistCp result has data and checksum mismatch when blocks per chunk > 0
• is related to: HADOOP-11794 Enable distcp to copy blocks in parallel
• is related to: HADOOP-16536 Backport HADOOP-16158 and HADOOP-15273 to branch-2
• links to: GitHub Pull Request #919