Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13163

Reuse pre-computed filestatus in Distcp-CopyMapper

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: tools/distcp
    • Labels:
      None

      Description

      https://github.com/apache/hadoop/blob/af942585a108d70e0946f6dd4c465a54d068eabf/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java#L185

      targetStatus is already computed and it can be reused in checkUpdate() function. This wouldn't be a major issue in NN/HDFS, but in the case of S3 getFileStatus calls can be expensive.

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9790 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9790/)
        HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh (stevel: rev c69a649257a331da55c1a1bf61c819e289015a6b)

        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9790 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9790/ ) HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh (stevel: rev c69a649257a331da55c1a1bf61c819e289015a6b) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java
        Hide
        stevel@apache.org Steve Loughran added a comment -

        +1

        patched -thanks!

        Show
        stevel@apache.org Steve Loughran added a comment - +1 patched -thanks!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 38s trunk passed
        +1 compile 0m 16s trunk passed
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 23s trunk passed
        +1 mvneclipse 0m 10s trunk passed
        +1 findbugs 0m 29s trunk passed
        +1 javadoc 0m 13s trunk passed
        +1 mvninstall 0m 18s the patch passed
        +1 compile 0m 16s the patch passed
        +1 javac 0m 16s the patch passed
        -1 checkstyle 0m 13s hadoop-tools/hadoop-distcp: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
        +1 mvnsite 0m 21s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 38s the patch passed
        +1 javadoc 0m 12s the patch passed
        +1 unit 8m 36s hadoop-distcp in the patch passed.
        +1 asflicense 0m 14s The patch does not generate ASF License warnings.
        22m 10s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804341/HADOOP-13163.001.patch
        JIRA Issue HADOOP-13163
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7016f84488fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9fe5828
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9457/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9457/testReport/
        modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9457/console
        Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 38s trunk passed +1 compile 0m 16s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvneclipse 0m 10s trunk passed +1 findbugs 0m 29s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 18s the patch passed +1 compile 0m 16s the patch passed +1 javac 0m 16s the patch passed -1 checkstyle 0m 13s hadoop-tools/hadoop-distcp: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19) +1 mvnsite 0m 21s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 38s the patch passed +1 javadoc 0m 12s the patch passed +1 unit 8m 36s hadoop-distcp in the patch passed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 22m 10s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804341/HADOOP-13163.001.patch JIRA Issue HADOOP-13163 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7016f84488fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9fe5828 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9457/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9457/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9457/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rajesh.balamohan Rajesh Balamohan added a comment -

        Attaching the patch for review. Existing test cases takes care of this code path.

        \cc Chris Nauroth, Steve Loughran

        Show
        rajesh.balamohan Rajesh Balamohan added a comment - Attaching the patch for review. Existing test cases takes care of this code path. \cc Chris Nauroth , Steve Loughran

          People

          • Assignee:
            rajesh.balamohan Rajesh Balamohan
            Reporter:
            rajesh.balamohan Rajesh Balamohan
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development