Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10313

Distcp need to enforce the order of snapshot names passed to -diff

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: distcp
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      This jira is to propose adding a check to distcp, when -diff s1 s2 is passed, we need to ensure that s2 is newer than s1, otherwise, abort with a informative error message.

      This is the result of my offline discussion with Jing Zhao on HDFS-9820. Thanks Jing.

      1. HDFS-10313.001.patch
        4 kB
        Yiqun Lin
      2. HDFS-10313.002.patch
        5 kB
        Yiqun Lin
      3. HDFS-10313.003.patch
        5 kB
        Yiqun Lin

        Issue Links

          Activity

          Hide
          linyiqun Yiqun Lin added a comment -

          Hi, Yongjun Zhang, I will upload a initial patch later.

          Show
          linyiqun Yiqun Lin added a comment - Hi, Yongjun Zhang , I will upload a initial patch later.
          Hide
          linyiqun Yiqun Lin added a comment -

          Attach a initial patch from me, thanks review.

          Show
          linyiqun Yiqun Lin added a comment - Attach a initial patch from me, thanks review.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 37s trunk passed
          +1 compile 0m 14s trunk passed with JDK v1.8.0_77
          +1 compile 0m 17s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 23s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 29s trunk passed
          +1 javadoc 0m 12s trunk passed with JDK v1.8.0_77
          +1 javadoc 0m 14s trunk passed with JDK v1.7.0_95
          -1 mvninstall 0m 11s hadoop-distcp in the patch failed.
          -1 compile 0m 8s hadoop-distcp in the patch failed with JDK v1.8.0_77.
          -1 javac 0m 8s hadoop-distcp in the patch failed with JDK v1.8.0_77.
          -1 compile 0m 11s hadoop-distcp in the patch failed with JDK v1.7.0_95.
          -1 javac 0m 11s hadoop-distcp in the patch failed with JDK v1.7.0_95.
          +1 checkstyle 0m 10s the patch passed
          -1 mvnsite 0m 12s hadoop-distcp in the patch failed.
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 0m 10s hadoop-distcp in the patch failed.
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_77
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95
          -1 unit 0m 9s hadoop-distcp in the patch failed with JDK v1.8.0_77.
          -1 unit 0m 11s hadoop-distcp in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          12m 3s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799664/HDFS-10313.001.patch
          JIRA Issue HDFS-10313
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f1f97aa008dc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / af9bdbe
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-distcp.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-distcp-jdk1.8.0_77.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-distcp-jdk1.8.0_77.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-distcp-jdk1.7.0_95.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-distcp-jdk1.7.0_95.txt
          mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-distcp.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-distcp.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15211/testReport/
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15211/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 37s trunk passed +1 compile 0m 14s trunk passed with JDK v1.8.0_77 +1 compile 0m 17s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 29s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 14s trunk passed with JDK v1.7.0_95 -1 mvninstall 0m 11s hadoop-distcp in the patch failed. -1 compile 0m 8s hadoop-distcp in the patch failed with JDK v1.8.0_77. -1 javac 0m 8s hadoop-distcp in the patch failed with JDK v1.8.0_77. -1 compile 0m 11s hadoop-distcp in the patch failed with JDK v1.7.0_95. -1 javac 0m 11s hadoop-distcp in the patch failed with JDK v1.7.0_95. +1 checkstyle 0m 10s the patch passed -1 mvnsite 0m 12s hadoop-distcp in the patch failed. +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 0m 10s hadoop-distcp in the patch failed. +1 javadoc 0m 10s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95 -1 unit 0m 9s hadoop-distcp in the patch failed with JDK v1.8.0_77. -1 unit 0m 11s hadoop-distcp in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 12m 3s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799664/HDFS-10313.001.patch JIRA Issue HDFS-10313 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f1f97aa008dc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / af9bdbe Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-distcp.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-distcp-jdk1.8.0_77.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-distcp-jdk1.8.0_77.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-distcp-jdk1.7.0_95.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-distcp-jdk1.7.0_95.txt mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-distcp.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-distcp.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15211/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15211/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15211/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          linyiqun Yiqun Lin added a comment -

          Sorry, the previous is not completed, update a latest patch.

          Show
          linyiqun Yiqun Lin added a comment - Sorry, the previous is not completed, update a latest patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 14s trunk passed
          +1 compile 0m 15s trunk passed with JDK v1.8.0_77
          +1 compile 0m 17s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 23s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 30s trunk passed
          +1 javadoc 0m 12s trunk passed with JDK v1.8.0_77
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95
          -1 mvninstall 0m 15s hadoop-distcp in the patch failed.
          -1 compile 0m 12s hadoop-distcp in the patch failed with JDK v1.8.0_77.
          -1 javac 0m 12s hadoop-distcp in the patch failed with JDK v1.8.0_77.
          -1 compile 0m 14s hadoop-distcp in the patch failed with JDK v1.7.0_95.
          -1 javac 0m 14s hadoop-distcp in the patch failed with JDK v1.7.0_95.
          +1 checkstyle 0m 10s the patch passed
          -1 mvnsite 0m 17s hadoop-distcp in the patch failed.
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 0m 11s hadoop-distcp in the patch failed.
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_77
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95
          -1 unit 0m 13s hadoop-distcp in the patch failed with JDK v1.8.0_77.
          -1 unit 0m 15s hadoop-distcp in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          13m 16s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799668/HDFS-10313.001.patch
          JIRA Issue HDFS-10313
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux fbecc92d6bf0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / af9bdbe
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-distcp.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-distcp-jdk1.8.0_77.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-distcp-jdk1.8.0_77.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-distcp-jdk1.7.0_95.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-distcp-jdk1.7.0_95.txt
          mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-distcp.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-distcp.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15212/testReport/
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15212/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 14s trunk passed +1 compile 0m 15s trunk passed with JDK v1.8.0_77 +1 compile 0m 17s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 30s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95 -1 mvninstall 0m 15s hadoop-distcp in the patch failed. -1 compile 0m 12s hadoop-distcp in the patch failed with JDK v1.8.0_77. -1 javac 0m 12s hadoop-distcp in the patch failed with JDK v1.8.0_77. -1 compile 0m 14s hadoop-distcp in the patch failed with JDK v1.7.0_95. -1 javac 0m 14s hadoop-distcp in the patch failed with JDK v1.7.0_95. +1 checkstyle 0m 10s the patch passed -1 mvnsite 0m 17s hadoop-distcp in the patch failed. +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 0m 11s hadoop-distcp in the patch failed. +1 javadoc 0m 10s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95 -1 unit 0m 13s hadoop-distcp in the patch failed with JDK v1.8.0_77. -1 unit 0m 15s hadoop-distcp in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 13m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799668/HDFS-10313.001.patch JIRA Issue HDFS-10313 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fbecc92d6bf0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / af9bdbe Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-distcp.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-distcp-jdk1.8.0_77.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-distcp-jdk1.8.0_77.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-distcp-jdk1.7.0_95.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-distcp-jdk1.7.0_95.txt mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-distcp.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-distcp.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15212/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15212/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15212/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 4s trunk passed
          +1 compile 0m 15s trunk passed with JDK v1.8.0_77
          +1 compile 0m 17s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 23s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 30s trunk passed
          +1 javadoc 0m 12s trunk passed with JDK v1.8.0_77
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 18s the patch passed
          +1 compile 0m 14s the patch passed with JDK v1.8.0_77
          +1 javac 0m 14s the patch passed
          +1 compile 0m 16s the patch passed with JDK v1.7.0_95
          +1 javac 0m 16s the patch passed
          +1 checkstyle 0m 11s the patch passed
          +1 mvnsite 0m 21s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 39s the patch passed
          +1 javadoc 0m 11s the patch passed with JDK v1.8.0_77
          +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95
          +1 unit 8m 34s hadoop-distcp in the patch passed with JDK v1.8.0_77.
          +1 unit 7m 35s hadoop-distcp in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          29m 36s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799675/HDFS-10313.001.patch
          JIRA Issue HDFS-10313
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 197e2099372f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / af9bdbe
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15213/testReport/
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15213/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 4s trunk passed +1 compile 0m 15s trunk passed with JDK v1.8.0_77 +1 compile 0m 17s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 30s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 18s the patch passed +1 compile 0m 14s the patch passed with JDK v1.8.0_77 +1 javac 0m 14s the patch passed +1 compile 0m 16s the patch passed with JDK v1.7.0_95 +1 javac 0m 16s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 21s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 39s the patch passed +1 javadoc 0m 11s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95 +1 unit 8m 34s hadoop-distcp in the patch passed with JDK v1.8.0_77. +1 unit 7m 35s hadoop-distcp in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 29m 36s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799675/HDFS-10313.001.patch JIRA Issue HDFS-10313 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 197e2099372f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / af9bdbe Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15213/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15213/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Yiqun Lin,

          Thanks a lot for working on this issue!

          I looked into your patch, it looks pretty good. I have a few comments, largely cosmetic things:

          1. It may be better to say "Snapshot <to> should be newer than <from>" in the following exception. Replace <from> <to> with the real names.

          throw new HadoopIllegalArgumentException(
          "The toSnapshot file should be newer than fromSnapshot file");

          2.

             } catch (FileNotFoundException nfe) {
                DistCp.LOG.warn("The snapshot file not be found.", nfe);
              }
          
          

          We should return false here. Or maybe we simply throw InvalidInputException, with nfe as the cause. I actually think the latter is better.

          3. In createAndSubmitJob() method,

                 if (distCpSync.sync()) {
                    createInputFileListingWithDiff(job, distCpSync);
                  } else {
                    inputOptions.disableUsingDiff();
                  }
          

          I'd suggest that in the else block, we don't disable using diff, and simply issue error with clear message, and throw InvalidInputException to be caught at DistCp#run method, thus quitting DistCp.

          Hi Jing Zhao, the old behavior is that we fallback to regular distcp if sync failed, my suggested change would change the old behavior so no fallback would happen, I believe it's safer, do you agree?

          4.

          public void testSyncOfTimeChecking() throws Exception {
          

          Suggest to change the test name to testSyncSnapshotTimeStampChecking()

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Yiqun Lin , Thanks a lot for working on this issue! I looked into your patch, it looks pretty good. I have a few comments, largely cosmetic things: 1. It may be better to say "Snapshot <to> should be newer than <from>" in the following exception. Replace <from> <to> with the real names. throw new HadoopIllegalArgumentException( "The toSnapshot file should be newer than fromSnapshot file"); 2. } catch (FileNotFoundException nfe) { DistCp.LOG.warn( "The snapshot file not be found." , nfe); } We should return false here. Or maybe we simply throw InvalidInputException , with nfe as the cause. I actually think the latter is better. 3. In createAndSubmitJob() method, if (distCpSync.sync()) { createInputFileListingWithDiff(job, distCpSync); } else { inputOptions.disableUsingDiff(); } I'd suggest that in the else block, we don't disable using diff, and simply issue error with clear message, and throw InvalidInputException to be caught at DistCp#run method, thus quitting DistCp. Hi Jing Zhao , the old behavior is that we fallback to regular distcp if sync failed, my suggested change would change the old behavior so no fallback would happen, I believe it's safer, do you agree? 4. public void testSyncOfTimeChecking() throws Exception { Suggest to change the test name to testSyncSnapshotTimeStampChecking() Thanks.
          Hide
          yzhangal Yongjun Zhang added a comment - - edited

          In addition, there is a method
          getSourceSnapshotPath(Path sourceDir, String snapshotName) you can use to replace two lines of code that compute snapshot path.

          Show
          yzhangal Yongjun Zhang added a comment - - edited In addition, there is a method getSourceSnapshotPath(Path sourceDir, String snapshotName) you can use to replace two lines of code that compute snapshot path.
          Hide
          jingzhao Jing Zhao added a comment -

          the old behavior is that we fallback to regular distcp if sync failed, my suggested change would change the old behavior so no fallback would happen, I believe it's safer, do you agree?

          Yes, agree. Let's return some error msg here.

          Show
          jingzhao Jing Zhao added a comment - the old behavior is that we fallback to regular distcp if sync failed, my suggested change would change the old behavior so no fallback would happen, I believe it's safer, do you agree? Yes, agree. Let's return some error msg here.
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Yongjun Zhang for review. Update the latest for addressing the comments.

          Show
          linyiqun Yiqun Lin added a comment - Thanks Yongjun Zhang for review. Update the latest for addressing the comments.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Yiqun Lin,

          Thanks for the new rev. A few more minor things below. I am +1 after they are addressed.

          1.

          117	      throw new InvalidInputException("Snapshot not be found: " + nfe);
          

          Add the following method in CopyListing.java

              public InvalidInputException(String message, Throwable cause) {
                super(message, cause);
              }    
          

          and change the call to

          117	      throw new InvalidInputException("Input snapshot is not found", nfe);
          

          2. In DistCp#createAndSubmitJob()

                   throw new InvalidInputException(
                        "Distcp sync failed, because of invalid options: " + inputOptions);
          

          DistCp sync may have failed for different reasons. Though I initially suggested InvalidInputException, I now think using Exception here is better. Sorry about that.

          3. Add spaces by changing

          731	    try{
          ...
          735	    }catch(HadoopIllegalArgumentException e){
          

          to

          731	    try {
          ...
          735	    } catch (HadoopIllegalArgumentException e) {
          

          Thanks.

          --Yongjun

          Show
          yzhangal Yongjun Zhang added a comment - Hi Yiqun Lin , Thanks for the new rev. A few more minor things below. I am +1 after they are addressed. 1. 117 throw new InvalidInputException( "Snapshot not be found: " + nfe); Add the following method in CopyListing.java public InvalidInputException( String message, Throwable cause) { super (message, cause); } and change the call to 117 throw new InvalidInputException( "Input snapshot is not found" , nfe); 2. In DistCp#createAndSubmitJob() throw new InvalidInputException( "Distcp sync failed, because of invalid options: " + inputOptions); DistCp sync may have failed for different reasons. Though I initially suggested InvalidInputException , I now think using Exception here is better. Sorry about that. 3. Add spaces by changing 731 try { ... 735 } catch (HadoopIllegalArgumentException e){ to 731 try { ... 735 } catch (HadoopIllegalArgumentException e) { Thanks. --Yongjun
          Hide
          linyiqun Yiqun Lin added a comment -

          Update the patch for the latest comments, pending jenkins.

          Show
          linyiqun Yiqun Lin added a comment - Update the patch for the latest comments, pending jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 32s trunk passed
          +1 compile 0m 14s trunk passed with JDK v1.8.0_77
          +1 compile 0m 17s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 22s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 28s trunk passed
          +1 javadoc 0m 12s trunk passed with JDK v1.8.0_77
          +1 javadoc 0m 14s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 17s the patch passed
          +1 compile 0m 12s the patch passed with JDK v1.8.0_77
          +1 javac 0m 12s the patch passed
          +1 compile 0m 15s the patch passed with JDK v1.7.0_95
          +1 javac 0m 15s the patch passed
          +1 checkstyle 0m 11s the patch passed
          +1 mvnsite 0m 20s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 38s the patch passed
          +1 javadoc 0m 9s the patch passed with JDK v1.8.0_77
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95
          +1 unit 7m 47s hadoop-distcp in the patch passed with JDK v1.8.0_77.
          +1 unit 7m 7s hadoop-distcp in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          27m 26s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12800165/HDFS-10313.003.patch
          JIRA Issue HDFS-10313
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 97089724b0fd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 337bcde
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15251/testReport/
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15251/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 32s trunk passed +1 compile 0m 14s trunk passed with JDK v1.8.0_77 +1 compile 0m 17s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 22s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 14s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 17s the patch passed +1 compile 0m 12s the patch passed with JDK v1.8.0_77 +1 javac 0m 12s the patch passed +1 compile 0m 15s the patch passed with JDK v1.7.0_95 +1 javac 0m 15s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 38s the patch passed +1 javadoc 0m 9s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95 +1 unit 7m 47s hadoop-distcp in the patch passed with JDK v1.8.0_77. +1 unit 7m 7s hadoop-distcp in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 27m 26s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12800165/HDFS-10313.003.patch JIRA Issue HDFS-10313 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 97089724b0fd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 337bcde Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15251/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15251/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          linyiqun Yiqun Lin added a comment -

          The latest patch looks good that all the unit tests are passed and there is no other checkstyle warnings. But I have fixed another minor thing in my patch, and update the v003 patch.

          Show
          linyiqun Yiqun Lin added a comment - The latest patch looks good that all the unit tests are passed and there is no other checkstyle warnings. But I have fixed another minor thing in my patch, and update the v003 patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 51s trunk passed
          +1 compile 0m 18s trunk passed with JDK v1.8.0_77
          +1 compile 0m 17s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 24s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 33s trunk passed
          +1 javadoc 0m 15s trunk passed with JDK v1.8.0_77
          +1 javadoc 0m 16s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 23s the patch passed
          +1 compile 0m 31s the patch passed with JDK v1.8.0_77
          +1 javac 0m 31s the patch passed
          +1 compile 0m 26s the patch passed with JDK v1.7.0_95
          +1 javac 0m 26s the patch passed
          +1 checkstyle 0m 14s the patch passed
          +1 mvnsite 0m 25s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 42s the patch passed
          +1 javadoc 0m 15s the patch passed with JDK v1.8.0_77
          +1 javadoc 0m 14s the patch passed with JDK v1.7.0_95
          +1 unit 11m 11s hadoop-distcp in the patch passed with JDK v1.8.0_77.
          +1 unit 11m 18s hadoop-distcp in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          38m 1s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12800360/HDFS-10313.003.patch
          JIRA Issue HDFS-10313
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b9ea05e32803 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 63e5412
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15262/testReport/
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15262/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 51s trunk passed +1 compile 0m 18s trunk passed with JDK v1.8.0_77 +1 compile 0m 17s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 24s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 33s trunk passed +1 javadoc 0m 15s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 16s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 23s the patch passed +1 compile 0m 31s the patch passed with JDK v1.8.0_77 +1 javac 0m 31s the patch passed +1 compile 0m 26s the patch passed with JDK v1.7.0_95 +1 javac 0m 26s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 42s the patch passed +1 javadoc 0m 15s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 14s the patch passed with JDK v1.7.0_95 +1 unit 11m 11s hadoop-distcp in the patch passed with JDK v1.8.0_77. +1 unit 11m 18s hadoop-distcp in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 38m 1s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12800360/HDFS-10313.003.patch JIRA Issue HDFS-10313 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b9ea05e32803 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 63e5412 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15262/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15262/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Thanks Yiqun Lin, +1 on rev 003. Will commit soon.

          Show
          yzhangal Yongjun Zhang added a comment - Thanks Yiqun Lin , +1 on rev 003. Will commit soon.
          Hide
          yzhangal Yongjun Zhang added a comment -

          I have committed to trunk, branch-2, branch-2.8. I also cherry-picked HDFS-10216 to branch-2.8 for cherry-pick cleanness.

          Thanks Yiqun Lin for the contribution, and Jing Zhao for comments.

          Show
          yzhangal Yongjun Zhang added a comment - I have committed to trunk, branch-2, branch-2.8. I also cherry-picked HDFS-10216 to branch-2.8 for cherry-pick cleanness. Thanks Yiqun Lin for the contribution, and Jing Zhao for comments.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9673 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9673/)
          HDFS-10313. Distcp need to enforce the order of snapshot names passed to (yzhang: rev 959a28dd1216dfac78d05b438828e8503108d963)

          • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
          • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListing.java
          • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java
          • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9673 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9673/ ) HDFS-10313 . Distcp need to enforce the order of snapshot names passed to (yzhang: rev 959a28dd1216dfac78d05b438828e8503108d963) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListing.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Yongjun Zhang for commit!

          Show
          linyiqun Yiqun Lin added a comment - Thanks Yongjun Zhang for commit!

            People

            • Assignee:
              linyiqun Yiqun Lin
              Reporter:
              yzhangal Yongjun Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development