Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13496

Include file lengths in Mismatch in length error for distcp

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2, 2.8.2
    • Component/s: None
    • Labels:

      Description

      Currently RetriableFileCopyCommand doesn't show the perceived lengths in Mismatch in length error:

      2016-08-12 10:23:14,231 ERROR [LocalJobRunner Map Task Executor #0] util.RetriableCommand(89): Failure in Retriable command: Copying hdfs://localhost:53941/user/tyu/test-data/dc7c674a-c463-4798-8260-   c5d1e3440a4b/WALs/10.22.9.171,53952,1471022508087/10.22.9.171%2C53952%2C1471022508087.regiongroup-1.1471022510182 to hdfs://localhost:53941/backupUT/backup_1471022580616/WALs/10.22.9.                   171%2C53952%2C1471022508087.regiongroup-1.1471022510182
      java.io.IOException: Mismatch in length of source:hdfs://localhost:53941/user/tyu/test-data/dc7c674a-c463-4798-8260-c5d1e3440a4b/WALs/10.22.9.171,53952,1471022508087/10.22.9.171%2C53952%2C1471022508087.regiongroup-1.1471022510182 and target:hdfs://localhost:53941/backupUT/backup_1471022580616/WALs/.distcp.tmp.attempt_local344329843_0006_m_000000_0
        at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareFileLengths(RetriableFileCopyCommand.java:193)
        at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:126)
        at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
        at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
        at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:281)
        at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:253)
        at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
      

      It would be helpful to include what's the expected length and what's the real length.

      Thanks to Yongjun Zhang for offline discussion.

        Attachments

          Activity

            People

            • Assignee:
              yuzhihong@gmail.com Ted Yu
              Reporter:
              yuzhihong@gmail.com Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: