Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14839

DistCp log output should contain copied and deleted files and directories

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: tools/distcp
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      DistCp log output (specified via -log command line option) currently contains only skipped and failed (when failures are ignored via -i) files.

      It will be more useful if it also contains copied and deleted files and created directories.

      This should be fixed in https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java

      1. HADOOP-14839.006.patch
        20 kB
        Yiqun Lin
      2. HADOOP-14839-branch-2.001.patch
        20 kB
        Yiqun Lin
      3. HADOOP-14839-branch-2.002.patch
        22 kB
        Yiqun Lin
      4. HDFS-10234.001.patch
        1 kB
        Yiqun Lin
      5. HDFS-10234.002.patch
        3 kB
        Yiqun Lin
      6. HDFS-10234.003.patch
        14 kB
        Yiqun Lin
      7. HDFS-10234.004.patch
        19 kB
        Yiqun Lin
      8. HDFS-10234.005.patch
        19 kB
        Yiqun Lin

        Activity

        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Yiqun Lin for the branch-2 patch. The patch looks good to me, +1.
        I manually tested it and committed to branch-2. The test failures are unrelated to this patch, I opened HADOOP-14860 to track the fix.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Yiqun Lin for the branch-2 patch. The patch looks good to me, +1. I manually tested it and committed to branch-2. The test failures are unrelated to this patch, I opened HADOOP-14860 to track the fix.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12811 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12811/)
        HADOOP-14839. DistCp log output should contain copied and deleted files (xyao: rev 63720ef574df6d79a9bab0628edb915d1e3a7c30)

        • (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java
        • (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        • (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
        • (edit) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpOptions.java
        • (edit) hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCp_Counter.properties
        • (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java
        • (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
        • (edit) hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm
        • (edit) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12811 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12811/ ) HADOOP-14839 . DistCp log output should contain copied and deleted files (xyao: rev 63720ef574df6d79a9bab0628edb915d1e3a7c30) (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java (edit) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpOptions.java (edit) hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCp_Counter.properties (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java (edit) hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm (edit) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
              branch-2 Compile Tests
        0 mvndep 0m 39s Maven dependency ordering for branch
        +1 mvninstall 6m 37s branch-2 passed
        +1 compile 1m 12s branch-2 passed with JDK v1.8.0_144
        +1 compile 1m 27s branch-2 passed with JDK v1.7.0_131
        +1 checkstyle 0m 30s branch-2 passed
        +1 mvnsite 0m 46s branch-2 passed
        +1 findbugs 1m 2s branch-2 passed
        +1 javadoc 0m 24s branch-2 passed with JDK v1.8.0_144
        +1 javadoc 0m 30s branch-2 passed with JDK v1.7.0_131
              Patch Compile Tests
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 0m 35s the patch passed
        +1 compile 1m 10s the patch passed with JDK v1.8.0_144
        +1 javac 1m 10s the patch passed
        +1 compile 1m 26s the patch passed with JDK v1.7.0_131
        +1 javac 1m 26s the patch passed
        +1 checkstyle 0m 29s the patch passed
        +1 mvnsite 0m 43s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 23s the patch passed
        +1 javadoc 0m 22s the patch passed with JDK v1.8.0_144
        +1 javadoc 0m 27s the patch passed with JDK v1.7.0_131
              Other Tests
        -1 unit 12m 44s hadoop-distcp in the patch failed with JDK v1.7.0_131.
        +1 unit 2m 13s hadoop-extras in the patch passed with JDK v1.7.0_131.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        58m 36s



        Reason Tests
        JDK v1.8.0_144 Failed junit tests hadoop.tools.TestIntegration
          hadoop.tools.TestDistCpViewFs
        JDK v1.7.0_131 Failed junit tests hadoop.tools.TestIntegration
          hadoop.tools.TestDistCpViewFs



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5e40efe
        JIRA Issue HADOOP-14839
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885583/HADOOP-14839-branch-2.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 9869fbf393cc 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 4c34c03
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13178/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13178/testReport/
        modules C: hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras U: hadoop-tools
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13178/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.       branch-2 Compile Tests 0 mvndep 0m 39s Maven dependency ordering for branch +1 mvninstall 6m 37s branch-2 passed +1 compile 1m 12s branch-2 passed with JDK v1.8.0_144 +1 compile 1m 27s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 30s branch-2 passed +1 mvnsite 0m 46s branch-2 passed +1 findbugs 1m 2s branch-2 passed +1 javadoc 0m 24s branch-2 passed with JDK v1.8.0_144 +1 javadoc 0m 30s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 0m 35s the patch passed +1 compile 1m 10s the patch passed with JDK v1.8.0_144 +1 javac 1m 10s the patch passed +1 compile 1m 26s the patch passed with JDK v1.7.0_131 +1 javac 1m 26s the patch passed +1 checkstyle 0m 29s the patch passed +1 mvnsite 0m 43s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 23s the patch passed +1 javadoc 0m 22s the patch passed with JDK v1.8.0_144 +1 javadoc 0m 27s the patch passed with JDK v1.7.0_131       Other Tests -1 unit 12m 44s hadoop-distcp in the patch failed with JDK v1.7.0_131. +1 unit 2m 13s hadoop-extras in the patch passed with JDK v1.7.0_131. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 58m 36s Reason Tests JDK v1.8.0_144 Failed junit tests hadoop.tools.TestIntegration   hadoop.tools.TestDistCpViewFs JDK v1.7.0_131 Failed junit tests hadoop.tools.TestIntegration   hadoop.tools.TestDistCpViewFs Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue HADOOP-14839 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885583/HADOOP-14839-branch-2.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9869fbf393cc 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 4c34c03 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13178/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13178/testReport/ modules C: hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras U: hadoop-tools Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13178/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        The failure test TestOptionsParser is related and other two can be passed in my local.
        Attach the updated patch of branch-2.

        Show
        linyiqun Yiqun Lin added a comment - The failure test TestOptionsParser is related and other two can be passed in my local. Attach the updated patch of branch-2.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              branch-2 Compile Tests
        0 mvndep 0m 41s Maven dependency ordering for branch
        +1 mvninstall 6m 49s branch-2 passed
        +1 compile 1m 19s branch-2 passed with JDK v1.8.0_144
        +1 compile 1m 32s branch-2 passed with JDK v1.7.0_131
        +1 checkstyle 0m 32s branch-2 passed
        +1 mvnsite 0m 47s branch-2 passed
        +1 findbugs 1m 1s branch-2 passed
        +1 javadoc 0m 27s branch-2 passed with JDK v1.8.0_144
        +1 javadoc 0m 29s branch-2 passed with JDK v1.7.0_131
              Patch Compile Tests
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 0m 37s the patch passed
        +1 compile 1m 15s the patch passed with JDK v1.8.0_144
        +1 javac 1m 15s the patch passed
        +1 compile 1m 32s the patch passed with JDK v1.7.0_131
        +1 javac 1m 32s the patch passed
        -0 checkstyle 0m 28s hadoop-tools: The patch generated 1 new + 211 unchanged - 0 fixed = 212 total (was 211)
        +1 mvnsite 0m 43s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 27s the patch passed
        +1 javadoc 0m 26s the patch passed with JDK v1.8.0_144
        +1 javadoc 0m 28s the patch passed with JDK v1.7.0_131
              Other Tests
        -1 unit 12m 58s hadoop-distcp in the patch failed with JDK v1.7.0_131.
        +1 unit 2m 6s hadoop-extras in the patch passed with JDK v1.7.0_131.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        60m 9s



        Reason Tests
        JDK v1.8.0_144 Failed junit tests hadoop.tools.TestOptionsParser
          hadoop.tools.TestIntegration
          hadoop.tools.TestDistCpViewFs
        JDK v1.7.0_131 Failed junit tests hadoop.tools.TestOptionsParser
          hadoop.tools.TestIntegration
          hadoop.tools.TestDistCpViewFs



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5e40efe
        JIRA Issue HADOOP-14839
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885559/HADOOP-14839-branch-2.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c873ba863c7f 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 4c34c03
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13177/artifact/patchprocess/diff-checkstyle-hadoop-tools.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13177/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13177/testReport/
        modules C: hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras U: hadoop-tools
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13177/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       branch-2 Compile Tests 0 mvndep 0m 41s Maven dependency ordering for branch +1 mvninstall 6m 49s branch-2 passed +1 compile 1m 19s branch-2 passed with JDK v1.8.0_144 +1 compile 1m 32s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 32s branch-2 passed +1 mvnsite 0m 47s branch-2 passed +1 findbugs 1m 1s branch-2 passed +1 javadoc 0m 27s branch-2 passed with JDK v1.8.0_144 +1 javadoc 0m 29s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 0m 37s the patch passed +1 compile 1m 15s the patch passed with JDK v1.8.0_144 +1 javac 1m 15s the patch passed +1 compile 1m 32s the patch passed with JDK v1.7.0_131 +1 javac 1m 32s the patch passed -0 checkstyle 0m 28s hadoop-tools: The patch generated 1 new + 211 unchanged - 0 fixed = 212 total (was 211) +1 mvnsite 0m 43s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 27s the patch passed +1 javadoc 0m 26s the patch passed with JDK v1.8.0_144 +1 javadoc 0m 28s the patch passed with JDK v1.7.0_131       Other Tests -1 unit 12m 58s hadoop-distcp in the patch failed with JDK v1.7.0_131. +1 unit 2m 6s hadoop-extras in the patch passed with JDK v1.7.0_131. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 60m 9s Reason Tests JDK v1.8.0_144 Failed junit tests hadoop.tools.TestOptionsParser   hadoop.tools.TestIntegration   hadoop.tools.TestDistCpViewFs JDK v1.7.0_131 Failed junit tests hadoop.tools.TestOptionsParser   hadoop.tools.TestIntegration   hadoop.tools.TestDistCpViewFs Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue HADOOP-14839 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885559/HADOOP-14839-branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c873ba863c7f 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 4c34c03 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13177/artifact/patchprocess/diff-checkstyle-hadoop-tools.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13177/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13177/testReport/ modules C: hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras U: hadoop-tools Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13177/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks Xiaoyu, attach the patch for branch-2.

        Show
        linyiqun Yiqun Lin added a comment - Thanks Xiaoyu, attach the patch for branch-2.
        Hide
        xyao Xiaoyu Yao added a comment -

        Yiqun Lin, can you submit a patch for branch-2 as this one can be a useful patch to back port?

        Show
        xyao Xiaoyu Yao added a comment - Yiqun Lin , can you submit a patch for branch-2 as this one can be a useful patch to back port?
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Yiqun Lin for the contribution and all for the reviews and discussions. I've commit the patch to trunk and branch-3.0.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Yiqun Lin for the contribution and all for the reviews and discussions. I've commit the patch to trunk and branch-3.0.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        0 mvndep 1m 51s Maven dependency ordering for branch
        +1 mvninstall 14m 14s trunk passed
        +1 compile 1m 29s trunk passed
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 0m 39s trunk passed
        +1 findbugs 0m 52s trunk passed
        +1 javadoc 0m 28s trunk passed
              Patch Compile Tests
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 0m 34s the patch passed
        +1 compile 1m 25s the patch passed
        +1 javac 1m 25s the patch passed
        +1 checkstyle 0m 25s the patch passed
        +1 mvnsite 0m 36s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 1s the patch passed
        +1 javadoc 0m 24s the patch passed
              Other Tests
        +1 unit 14m 42s hadoop-distcp in the patch passed.
        +1 unit 0m 46s hadoop-extras in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        47m 18s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HADOOP-14839
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885504/HADOOP-14839.006.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b06f25456418 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d4035d4
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13175/testReport/
        modules C: hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras U: hadoop-tools
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13175/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 1m 51s Maven dependency ordering for branch +1 mvninstall 14m 14s trunk passed +1 compile 1m 29s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 39s trunk passed +1 findbugs 0m 52s trunk passed +1 javadoc 0m 28s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 34s the patch passed +1 compile 1m 25s the patch passed +1 javac 1m 25s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 1s the patch passed +1 javadoc 0m 24s the patch passed       Other Tests +1 unit 14m 42s hadoop-distcp in the patch passed. +1 unit 0m 46s hadoop-extras in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 47m 18s Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HADOOP-14839 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885504/HADOOP-14839.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b06f25456418 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d4035d4 Default Java 1.8.0_144 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13175/testReport/ modules C: hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras U: hadoop-tools Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13175/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks Xiaoyu Yao for catching this, Update the updated patch.

        Show
        linyiqun Yiqun Lin added a comment - Thanks Xiaoyu Yao for catching this, Update the updated patch.
        Hide
        xyao Xiaoyu Yao added a comment -

        Also, move the JIRA from hadoop-hdfs to hadoop-common project. Please change the patch name accordingly when attaching new patch.

        Show
        xyao Xiaoyu Yao added a comment - Also, move the JIRA from hadoop-hdfs to hadoop-common project. Please change the patch name accordingly when attaching new patch.
        Hide
        xyao Xiaoyu Yao added a comment -

        Yiqun Lin, can you update the DistCp_Counter.properties with the new DIR_COPY counter? Without it, the DistCp Counters output will only give raw name like below, which is not very user friendly.

        	DistCp Counters
        ...
                        Files Copied=6
        		DIR_COPY=3
        
        Show
        xyao Xiaoyu Yao added a comment - Yiqun Lin , can you update the DistCp_Counter.properties with the new DIR_COPY counter? Without it, the DistCp Counters output will only give raw name like below, which is not very user friendly. DistCp Counters ... Files Copied=6 DIR_COPY=3
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 15m 26s trunk passed
        +1 compile 0m 21s trunk passed
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 25s trunk passed
        +1 findbugs 0m 28s trunk passed
        +1 javadoc 0m 14s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 18s the patch passed
        +1 compile 0m 19s the patch passed
        +1 javac 0m 19s the patch passed
        +1 checkstyle 0m 14s the patch passed
        +1 mvnsite 0m 21s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 33s the patch passed
        +1 javadoc 0m 11s the patch passed
              Other Tests
        +1 unit 14m 0s hadoop-distcp in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        34m 57s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HADOOP-14839
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884459/HDFS-10234.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7a10693c26d6 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0ba8ff4
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13171/testReport/
        modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13171/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 15m 26s trunk passed +1 compile 0m 21s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 25s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 14s trunk passed       Patch Compile Tests +1 mvninstall 0m 18s the patch passed +1 compile 0m 19s the patch passed +1 javac 0m 19s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 33s the patch passed +1 javadoc 0m 11s the patch passed       Other Tests +1 unit 14m 0s hadoop-distcp in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 34m 57s Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HADOOP-14839 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884459/HDFS-10234.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7a10693c26d6 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0ba8ff4 Default Java 1.8.0_144 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13171/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13171/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Yiqun Lin for the update. Patch v5 looks good to me. +1.
        I will hold off the commit until Friday (9/1) in case Konstantin Shaposhnikov and/or other folks may have additional comments.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Yiqun Lin for the update. Patch v5 looks good to me. +1. I will hold off the commit until Friday (9/1) in case Konstantin Shaposhnikov and/or other folks may have additional comments.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 34s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 15m 31s trunk passed
        +1 compile 0m 22s trunk passed
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 23s trunk passed
        +1 findbugs 0m 30s trunk passed
        +1 javadoc 0m 17s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 20s the patch passed
        +1 compile 0m 18s the patch passed
        +1 javac 0m 18s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 20s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 34s the patch passed
        +1 javadoc 0m 14s the patch passed
              Other Tests
        +1 unit 15m 9s hadoop-distcp in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        36m 9s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-10234
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884459/HDFS-10234.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b36ed5c9b44c 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 200b113
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20925/testReport/
        modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20925/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 34s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 15m 31s trunk passed +1 compile 0m 22s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 23s trunk passed +1 findbugs 0m 30s trunk passed +1 javadoc 0m 17s trunk passed       Patch Compile Tests +1 mvninstall 0m 20s the patch passed +1 compile 0m 18s the patch passed +1 javac 0m 18s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 34s the patch passed +1 javadoc 0m 14s the patch passed       Other Tests +1 unit 15m 9s hadoop-distcp in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 36m 9s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-10234 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884459/HDFS-10234.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b36ed5c9b44c 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 200b113 Default Java 1.8.0_144 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20925/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20925/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        Failed test is related. Attach the new patch to fix this.

        Show
        linyiqun Yiqun Lin added a comment - Failed test is related. Attach the new patch to fix this.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 15m 56s trunk passed
        +1 compile 0m 21s trunk passed
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 24s trunk passed
        +1 findbugs 0m 28s trunk passed
        +1 javadoc 0m 14s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 19s the patch passed
        +1 compile 0m 17s the patch passed
        +1 javac 0m 17s the patch passed
        +1 checkstyle 0m 12s the patch passed
        +1 mvnsite 0m 21s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 39s the patch passed
        +1 javadoc 0m 12s the patch passed
              Other Tests
        -1 unit 14m 15s hadoop-distcp in the patch failed.
        +1 asflicense 0m 14s The patch does not generate ASF License warnings.
        35m 7s



        Reason Tests
        Failed junit tests hadoop.tools.TestDistCpOptions



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-10234
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884452/HDFS-10234.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 4bfdb7a09512 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 200b113
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20923/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20923/testReport/
        modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20923/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 15m 56s trunk passed +1 compile 0m 21s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 24s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 14s trunk passed       Patch Compile Tests +1 mvninstall 0m 19s the patch passed +1 compile 0m 17s the patch passed +1 javac 0m 17s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 39s the patch passed +1 javadoc 0m 12s the patch passed       Other Tests -1 unit 14m 15s hadoop-distcp in the patch failed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 35m 7s Reason Tests Failed junit tests hadoop.tools.TestDistCpOptions Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-10234 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884452/HDFS-10234.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4bfdb7a09512 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 200b113 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/20923/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20923/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20923/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks for the review, Xiaoyu Yao!
        Attach the updated patch to address your comments.

        Show
        linyiqun Yiqun Lin added a comment - Thanks for the review, Xiaoyu Yao ! Attach the updated patch to address your comments.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Yiqun Lin for the update. The patch v3 looks pretty good to me. I just have few comments below:

        DistCpOptionSwitch.java
        Line 125: NIT: "Log additional info (path, size) in the SKIP/COPY log."

        CopyMapper.java
        Line 86: NIT: verboseWrite -> verboseLog
        Line 110: Can we check and ensure verboseWrite can only be used with -log option?

        DistCp.md.vm
        Line 223: `-v` | Log additional info (path, size) in the SKIP/COPY log | This option can
        only be used with -log option.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Yiqun Lin for the update. The patch v3 looks pretty good to me. I just have few comments below: DistCpOptionSwitch.java Line 125: NIT: "Log additional info (path, size) in the SKIP/COPY log." CopyMapper.java Line 86: NIT: verboseWrite -> verboseLog Line 110: Can we check and ensure verboseWrite can only be used with -log option? DistCp.md.vm Line 223: `-v` | Log additional info (path, size) in the SKIP/COPY log | This option can only be used with -log option.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 49s trunk passed
        +1 compile 0m 20s trunk passed
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 21s trunk passed
        +1 findbugs 0m 25s trunk passed
        +1 javadoc 0m 14s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 17s the patch passed
        +1 compile 0m 16s the patch passed
        +1 javac 0m 16s the patch passed
        -0 checkstyle 0m 13s hadoop-tools/hadoop-distcp: The patch generated 1 new + 139 unchanged - 0 fixed = 140 total (was 139)
        +1 mvnsite 0m 18s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 30s the patch passed
        +1 javadoc 0m 10s the patch passed
              Other Tests
        +1 unit 13m 54s hadoop-distcp in the patch passed.
        +1 asflicense 0m 14s The patch does not generate ASF License warnings.
        32m 10s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-10234
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883989/HDFS-10234.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux ae06e9330b77 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ae7abad
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20885/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20885/testReport/
        modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20885/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 49s trunk passed +1 compile 0m 20s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 21s trunk passed +1 findbugs 0m 25s trunk passed +1 javadoc 0m 14s trunk passed       Patch Compile Tests +1 mvninstall 0m 17s the patch passed +1 compile 0m 16s the patch passed +1 javac 0m 16s the patch passed -0 checkstyle 0m 13s hadoop-tools/hadoop-distcp: The patch generated 1 new + 139 unchanged - 0 fixed = 140 total (was 139) +1 mvnsite 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 30s the patch passed +1 javadoc 0m 10s the patch passed       Other Tests +1 unit 13m 54s hadoop-distcp in the patch passed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 32m 10s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-10234 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883989/HDFS-10234.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ae06e9330b77 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ae7abad Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20885/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20885/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20885/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        Attach the updated patch to address the review comment. Please have a review, Xiaoyu Yao.Thanks.

        Show
        linyiqun Yiqun Lin added a comment - Attach the updated patch to address the review comment. Please have a review, Xiaoyu Yao .Thanks.
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks for the review, Xiaoyu!
        Seems this JIRA hasn't been updated more than one year, I will spend some time to look into this again recent days, . Thanks.

        Show
        linyiqun Yiqun Lin added a comment - Thanks for the review, Xiaoyu! Seems this JIRA hasn't been updated more than one year, I will spend some time to look into this again recent days, . Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 6s HDFS-10234 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue HDFS-10234
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796999/HDFS-10234.002.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20831/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s HDFS-10234 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-10234 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796999/HDFS-10234.002.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20831/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xyao Xiaoyu Yao added a comment - - edited

        Yiqun Lin, thanks for the patch. I take a look at the v2 patch here are my comments:

        1. We need replace the COPY count update in createTargetDirsWithRetry with a DIR_COPY count update. The change to update DIR_COPY counter in map() can be removed after that. Without that, Konstantin Shaposhnikov's earlier comment are not fully addressed.

        @@ -260,7 +268,7 @@ private void createTargetDirsWithRetry(String description,
             } catch (Exception e) {
               throw new IOException("mkdir failed for " + target, e);
             }
        -    incrementCounter(context, Counter.COPY, 1);
        +    incrementCounter(context, Counter.DIR_COPY, 1);
           }
        

        2. Can we include both the source (path, size) and destination (path, size) in the SKIP/COPY log inside map()? The information is available from sourceCurrStatus and targetStatus there. This way, many applications can just parse the distcp log offline to get information without adding extra load on namenode.

        3. We will need a switch (e.g., -v) to enable these additional log output for backward compatibility. By default, the log only contains the information as it is today.

        Show
        xyao Xiaoyu Yao added a comment - - edited Yiqun Lin , thanks for the patch. I take a look at the v2 patch here are my comments: 1. We need replace the COPY count update in createTargetDirsWithRetry with a DIR_COPY count update. The change to update DIR_COPY counter in map() can be removed after that. Without that, Konstantin Shaposhnikov 's earlier comment are not fully addressed. @@ -260,7 +268,7 @@ private void createTargetDirsWithRetry( String description, } catch (Exception e) { throw new IOException( "mkdir failed for " + target, e); } - incrementCounter(context, Counter.COPY, 1); + incrementCounter(context, Counter.DIR_COPY, 1); } 2. Can we include both the source (path, size) and destination (path, size) in the SKIP/COPY log inside map()? The information is available from sourceCurrStatus and targetStatus there. This way, many applications can just parse the distcp log offline to get information without adding extra load on namenode. 3. We will need a switch (e.g., -v) to enable these additional log output for backward compatibility. By default, the log only contains the information as it is today.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 32s Maven dependency ordering for branch
        +1 mvninstall 7m 17s trunk passed
        +1 compile 1m 24s trunk passed with JDK v1.8.0_77
        +1 compile 1m 23s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 27s trunk passed
        +1 mvnsite 0m 43s trunk passed
        +1 mvneclipse 0m 28s trunk passed
        +1 findbugs 0m 58s trunk passed
        +1 javadoc 0m 26s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 29s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 12s Maven dependency ordering for patch
        +1 mvninstall 0m 33s the patch passed
        +1 compile 1m 22s the patch passed with JDK v1.8.0_77
        +1 javac 1m 22s the patch passed
        +1 compile 1m 26s the patch passed with JDK v1.7.0_95
        +1 javac 1m 26s the patch passed
        +1 checkstyle 0m 24s the patch passed
        +1 mvnsite 0m 41s the patch passed
        +1 mvneclipse 0m 24s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 18s the patch passed
        +1 javadoc 0m 26s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95
        -1 unit 8m 43s hadoop-distcp in the patch failed with JDK v1.8.0_77.
        +1 unit 0m 58s hadoop-extras in the patch passed with JDK v1.8.0_77.
        -1 unit 8m 37s hadoop-distcp in the patch failed with JDK v1.7.0_95.
        +1 unit 0m 55s hadoop-extras in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        42m 16s



        Reason Tests
        JDK v1.8.0_77 Failed junit tests hadoop.tools.mapred.TestCopyMapper
        JDK v1.7.0_95 Failed junit tests hadoop.tools.mapred.TestCopyMapper



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796999/HDFS-10234.002.patch
        JIRA Issue HDFS-10234
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f31b6e9a1f4a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 818d6b7
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15066/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.8.0_77.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15066/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15066/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15066/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15066/testReport/
        modules C: hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras U: hadoop-tools
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15066/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 32s Maven dependency ordering for branch +1 mvninstall 7m 17s trunk passed +1 compile 1m 24s trunk passed with JDK v1.8.0_77 +1 compile 1m 23s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 43s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 0m 58s trunk passed +1 javadoc 0m 26s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 29s trunk passed with JDK v1.7.0_95 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 0m 33s the patch passed +1 compile 1m 22s the patch passed with JDK v1.8.0_77 +1 javac 1m 22s the patch passed +1 compile 1m 26s the patch passed with JDK v1.7.0_95 +1 javac 1m 26s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 41s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 18s the patch passed +1 javadoc 0m 26s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95 -1 unit 8m 43s hadoop-distcp in the patch failed with JDK v1.8.0_77. +1 unit 0m 58s hadoop-extras in the patch passed with JDK v1.8.0_77. -1 unit 8m 37s hadoop-distcp in the patch failed with JDK v1.7.0_95. +1 unit 0m 55s hadoop-extras in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 42m 16s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.tools.mapred.TestCopyMapper JDK v1.7.0_95 Failed junit tests hadoop.tools.mapred.TestCopyMapper Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796999/HDFS-10234.002.patch JIRA Issue HDFS-10234 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f31b6e9a1f4a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 818d6b7 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15066/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15066/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15066/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15066/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15066/testReport/ modules C: hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras U: hadoop-tools Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15066/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks Konstantin Shaposhnikov for concrete descriptions. Update the latest patch for addressing your comments.

        Show
        linyiqun Yiqun Lin added a comment - Thanks Konstantin Shaposhnikov for concrete descriptions. Update the latest patch for addressing your comments.
        Hide
        k.shaposhnikov@gmail.com Konstantin Shaposhnikov added a comment -

        Another way to make the job output more useful is to have separate counters for copied files and directories. Currently even if a directory exists on target COPY counter is incremented. So if I run distcp with -update option it still reports large number in COPY counter even though the number of copied files is 0.

        Show
        k.shaposhnikov@gmail.com Konstantin Shaposhnikov added a comment - Another way to make the job output more useful is to have separate counters for copied files and directories. Currently even if a directory exists on target COPY counter is incremented. So if I run distcp with -update option it still reports large number in COPY counter even though the number of copied files is 0.
        Hide
        k.shaposhnikov@gmail.com Konstantin Shaposhnikov added a comment -

        I guess my problem description was not very precise.

        The log file that I am interested in is created when `-log logDir` option is specified. This file is written using context.write (e.g. context.write(null, new Text("SKIP: " + sourceCurrStatus.getPath()));).

        Currently this file contains only SKIP entries making it almost useless.

        There is no need to make logging on INFO level more verbose as it is already too verbose.

        Show
        k.shaposhnikov@gmail.com Konstantin Shaposhnikov added a comment - I guess my problem description was not very precise. The log file that I am interested in is created when `-log logDir` option is specified. This file is written using context.write (e.g. context.write(null, new Text("SKIP: " + sourceCurrStatus.getPath())); ). Currently this file contains only SKIP entries making it almost useless. There is no need to make logging on INFO level more verbose as it is already too verbose.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 26s trunk passed
        +1 compile 0m 14s trunk passed with JDK v1.8.0_74
        +1 compile 0m 17s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 22s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 28s trunk passed
        +1 javadoc 0m 12s trunk passed with JDK v1.8.0_74
        +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 18s the patch passed
        +1 compile 0m 11s the patch passed with JDK v1.8.0_74
        +1 javac 0m 11s the patch passed
        +1 compile 0m 14s the patch passed with JDK v1.7.0_95
        +1 javac 0m 14s the patch passed
        +1 checkstyle 0m 11s the patch passed
        +1 mvnsite 0m 20s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 38s the patch passed
        +1 javadoc 0m 9s the patch passed with JDK v1.8.0_74
        +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95
        +1 unit 7m 23s hadoop-distcp in the patch passed with JDK v1.8.0_74.
        +1 unit 7m 16s hadoop-distcp in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 16s Patch does not generate ASF License warnings.
        27m 6s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796266/HDFS-10234.001.patch
        JIRA Issue HDFS-10234
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 52293d3b477c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0064cba
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15024/testReport/
        modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15024/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 26s trunk passed +1 compile 0m 14s trunk passed with JDK v1.8.0_74 +1 compile 0m 17s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 22s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 18s the patch passed +1 compile 0m 11s the patch passed with JDK v1.8.0_74 +1 javac 0m 11s the patch passed +1 compile 0m 14s the patch passed with JDK v1.7.0_95 +1 javac 0m 14s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 38s the patch passed +1 javadoc 0m 9s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95 +1 unit 7m 23s hadoop-distcp in the patch passed with JDK v1.8.0_74. +1 unit 7m 16s hadoop-distcp in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 16s Patch does not generate ASF License warnings. 27m 6s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796266/HDFS-10234.001.patch JIRA Issue HDFS-10234 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 52293d3b477c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0064cba Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15024/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15024/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        Upload a simple patch from me. I added the log output info for copying and and creating directories. But it seems there was no place indicated that the file is deleted in class CopyMapper. Can this patch satisfied you. If I was missing something, you can tell me.

        Show
        linyiqun Yiqun Lin added a comment - Upload a simple patch from me. I added the log output info for copying and and creating directories. But it seems there was no place indicated that the file is deleted in class CopyMapper . Can this patch satisfied you. If I was missing something, you can tell me.

          People

          • Assignee:
            linyiqun Yiqun Lin
            Reporter:
            k.shaposhnikov@gmail.com Konstantin Shaposhnikov
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development