Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13496

Include file lengths in Mismatch in length error for distcp

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2, 2.8.2
    • Component/s: None
    • Labels:

      Description

      Currently RetriableFileCopyCommand doesn't show the perceived lengths in Mismatch in length error:

      2016-08-12 10:23:14,231 ERROR [LocalJobRunner Map Task Executor #0] util.RetriableCommand(89): Failure in Retriable command: Copying hdfs://localhost:53941/user/tyu/test-data/dc7c674a-c463-4798-8260-   c5d1e3440a4b/WALs/10.22.9.171,53952,1471022508087/10.22.9.171%2C53952%2C1471022508087.regiongroup-1.1471022510182 to hdfs://localhost:53941/backupUT/backup_1471022580616/WALs/10.22.9.                   171%2C53952%2C1471022508087.regiongroup-1.1471022510182
      java.io.IOException: Mismatch in length of source:hdfs://localhost:53941/user/tyu/test-data/dc7c674a-c463-4798-8260-c5d1e3440a4b/WALs/10.22.9.171,53952,1471022508087/10.22.9.171%2C53952%2C1471022508087.regiongroup-1.1471022510182 and target:hdfs://localhost:53941/backupUT/backup_1471022580616/WALs/.distcp.tmp.attempt_local344329843_0006_m_000000_0
        at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareFileLengths(RetriableFileCopyCommand.java:193)
        at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:126)
        at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
        at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
        at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:281)
        at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:253)
        at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
      

      It would be helpful to include what's the expected length and what's the real length.

      Thanks to Yongjun Zhang for offline discussion.

      1. HADOOP-13496.v1.patch
        1 kB
        Ted Yu
      2. HADOOP-13496.v1.patch
        1 kB
        Ted Yu

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 40s trunk passed
        +1 compile 0m 16s trunk passed
        +1 checkstyle 0m 13s trunk passed
        +1 mvnsite 0m 20s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 24s trunk passed
        +1 javadoc 0m 13s trunk passed
        +1 mvninstall 0m 15s the patch passed
        +1 compile 0m 14s the patch passed
        +1 javac 0m 14s the patch passed
        -0 checkstyle 0m 10s hadoop-tools/hadoop-distcp: The patch generated 2 new + 10 unchanged - 1 fixed = 12 total (was 11)
        +1 mvnsite 0m 18s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 28s the patch passed
        +1 javadoc 0m 10s the patch passed
        +1 unit 9m 6s hadoop-distcp in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        20m 58s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-13496
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823506/HADOOP-13496.v1.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 58f6a6bdd055 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e1ad598
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10400/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10400/testReport/
        modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10400/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 40s trunk passed +1 compile 0m 16s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 24s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 15s the patch passed +1 compile 0m 14s the patch passed +1 javac 0m 14s the patch passed -0 checkstyle 0m 10s hadoop-tools/hadoop-distcp: The patch generated 2 new + 10 unchanged - 1 fixed = 12 total (was 11) +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 28s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 9m 6s hadoop-distcp in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 20m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13496 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823506/HADOOP-13496.v1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 58f6a6bdd055 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e1ad598 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10400/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10400/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10400/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -

        diff-checkstyle-hadoop-tools_hadoop-distcp.txt is no longer accessible.

        Show
        yuzhihong@gmail.com Ted Yu added a comment - diff-checkstyle-hadoop-tools_hadoop-distcp.txt is no longer accessible.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 5m 30s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 9m 47s trunk passed
        +1 compile 0m 21s trunk passed
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 24s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 0m 31s trunk passed
        +1 javadoc 0m 14s trunk passed
        +1 mvninstall 0m 20s the patch passed
        +1 compile 0m 20s the patch passed
        +1 javac 0m 20s the patch passed
        -0 checkstyle 0m 13s hadoop-tools/hadoop-distcp: The patch generated 2 new + 9 unchanged - 1 fixed = 11 total (was 10)
        +1 mvnsite 0m 23s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 40s the patch passed
        +1 javadoc 0m 12s the patch passed
        +1 unit 11m 55s hadoop-distcp in the patch passed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        33m 24s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HADOOP-13496
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839075/HADOOP-13496.v1.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 6fd598d28c83 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f121d0b
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11071/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11071/testReport/
        modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11071/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 5m 30s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 9m 47s trunk passed +1 compile 0m 21s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 24s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 31s trunk passed +1 javadoc 0m 14s trunk passed +1 mvninstall 0m 20s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed -0 checkstyle 0m 13s hadoop-tools/hadoop-distcp: The patch generated 2 new + 9 unchanged - 1 fixed = 11 total (was 10) +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 40s the patch passed +1 javadoc 0m 12s the patch passed +1 unit 11m 55s hadoop-distcp in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 33m 24s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13496 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839075/HADOOP-13496.v1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6fd598d28c83 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f121d0b Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11071/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11071/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11071/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        I'd completely missed this, sorry.

        1. is it still current
        2. which versions does it affect & for which the patch is targed to?

        I'd like the getFileStatus to be try/caught, so that if a problem arises there it doesn't get in the way of the root cause.

        Show
        stevel@apache.org Steve Loughran added a comment - I'd completely missed this, sorry. is it still current which versions does it affect & for which the patch is targed to? I'd like the getFileStatus to be try/caught, so that if a problem arises there it doesn't get in the way of the root cause.
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -

        The patch applies cleanly on latest trunk.

        which versions does it affect

        2.7.3+

        for which the patch is targed to

        trunk and branch-2

        Show
        yuzhihong@gmail.com Ted Yu added a comment - The patch applies cleanly on latest trunk. which versions does it affect 2.7.3+ for which the patch is targed to trunk and branch-2
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -

        Steve Loughran:
        Waiting for your feedback.

        Show
        yuzhihong@gmail.com Ted Yu added a comment - Steve Loughran : Waiting for your feedback.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        oh, almost good to go. One change

             } catch (Exception ex) {
        -      throw new ServiceFailedException(ex.getMessage());
        +      throw new ServiceFailedException("RefreshAll operation failed", ex);
             }
        

        can you make it ("RefreshAll operation failed: " + ex, ex). Stops the underlying cause being lost in the traces; handles exceptions with null getMessage() value, e.g. NPE

        Show
        stevel@apache.org Steve Loughran added a comment - oh, almost good to go. One change } catch (Exception ex) { - throw new ServiceFailedException(ex.getMessage()); + throw new ServiceFailedException( "RefreshAll operation failed" , ex); } can you make it ("RefreshAll operation failed: " + ex, ex). Stops the underlying cause being lost in the traces; handles exceptions with null getMessage() value, e.g. NPE
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -

        The above snippet was not in my patch.

        I checked hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java which doesn't have "ServiceFailedException(ex.getMessage())" either.

        Please clarify.

        Show
        yuzhihong@gmail.com Ted Yu added a comment - The above snippet was not in my patch. I checked hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java which doesn't have "ServiceFailedException(ex.getMessage())" either. Please clarify.
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -
        Show
        yuzhihong@gmail.com Ted Yu added a comment - Ping Steve Loughran
        Hide
        stevel@apache.org Steve Loughran added a comment -

        sorry, must have applied it to branch-2 with some other applied patch in, then looked @ the diff

        Show
        stevel@apache.org Steve Loughran added a comment - sorry, must have applied it to branch-2 with some other applied patch in, then looked @ the diff
        Hide
        stevel@apache.org Steve Loughran added a comment -

        1, committed to branch 2.8

        Show
        stevel@apache.org Steve Loughran added a comment - 1, committed to branch 2.8
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11144 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11144/)
        HADOOP-13496. Include file lengths in Mismatch in length error for (stevel: rev ed33ce11dd8de36fb79e103d8491d077cd4aaf77)

        • (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11144 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11144/ ) HADOOP-13496 . Include file lengths in Mismatch in length error for (stevel: rev ed33ce11dd8de36fb79e103d8491d077cd4aaf77) (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          People

          • Assignee:
            yuzhihong@gmail.com Ted Yu
            Reporter:
            yuzhihong@gmail.com Ted Yu
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development