[HADOOP-17932] Distcp file length comparison have no effect

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.1
• Fix Version/s: 3.4.0, 3.3.2
• Component/s: common, tools, tools/distcp
• Labels: pull-request-available
• External issue URL: https://github.com/apache/hadoop/pull/3519
• Assignee: yinan zhan
• Reporter: yinan zhan
• Created: 23/Sep/21 06:14
• Updated: 19/Oct/21 09:31
• Resolved: 18/Oct/21 10:22

Description

the params for compareFileLengthsAndChecksums in RetriableFileCopyCommand have no effect

current is

        DistCpUtils.compareFileLengthsAndChecksums(source.getLen(), sourceFS,
                sourcePath, sourceChecksum, targetFS,
                targetPath, skipCrc, source.getLen());

public static void compareFileLengthsAndChecksums(long srcLen,
           FileSystem sourceFS, Path source, FileChecksum sourceChecksum,
           FileSystem targetFS, Path target, boolean skipCrc,
           long targetLen) throws IOException {
  if (srcLen != targetLen) {
    throw new IOException(
        DistCpConstants.LENGTH_MISMATCH_ERROR_MSG + source + " (" + srcLen
            + ") and target:" + target + " (" + targetLen + ")");
  }

so, compare source.getLen() with source.getLen()...

It should be like below in history view

        DistCpUtils.compareFileLengthsAndChecksums(source.getLen(), sourceFS,
                sourcePath, sourceChecksum, targetFS,
                targetPath, skipCrc, offset + bytesRead);

 

 

Issue Links

• links to: GitHub Pull Request #3519