Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8151

Always use snapshot path as source when invalid snapshot names are used for diff based distcp

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: distcp
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This is a bug reported by Sushmitha Sreenivasan:

      HDFS-8036 makes the diff-based distcp use snapshot path as the source. This should also happen when

      1. invalid snapshot names are provided as distcp parameters thus the diff report computation on the target cluster fails
      2. there is modification happening in the target cluster thus checkNoChange returns false

      In other cases like source and target FS are not DistributedFileSystem, we should throw exceptions.

        Activity

        Hide
        jingzhao Jing Zhao added a comment -

        Patch to fix.

        Show
        jingzhao Jing Zhao added a comment - Patch to fix.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12725650/HDFS-8151.000.patch
        against trunk revision 9e8309a.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-distcp.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10282//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10282//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12725650/HDFS-8151.000.patch against trunk revision 9e8309a. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-distcp. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10282//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10282//console This message is automatically generated.
        Hide
        jingzhao Jing Zhao added a comment -

        Thanks for the review, Nicholas! I've committed this.

        Show
        jingzhao Jing Zhao added a comment - Thanks for the review, Nicholas! I've committed this.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7591/)
        HDFS-8151. Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. (jing9: rev 4c097e473bb1f18d1510deb61bae2bcb8c156f18)

        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7591/ ) HDFS-8151 . Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. (jing9: rev 4c097e473bb1f18d1510deb61bae2bcb8c156f18) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2097 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2097/)
        HDFS-8151. Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. (jing9: rev 4c097e473bb1f18d1510deb61bae2bcb8c156f18)

        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2097 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2097/ ) HDFS-8151 . Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. (jing9: rev 4c097e473bb1f18d1510deb61bae2bcb8c156f18) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #156 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/156/)
        HDFS-8151. Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. (jing9: rev 4c097e473bb1f18d1510deb61bae2bcb8c156f18)

        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #156 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/156/ ) HDFS-8151 . Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. (jing9: rev 4c097e473bb1f18d1510deb61bae2bcb8c156f18) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #165 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/165/)
        HDFS-8151. Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. (jing9: rev 4c097e473bb1f18d1510deb61bae2bcb8c156f18)

        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #165 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/165/ ) HDFS-8151 . Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. (jing9: rev 4c097e473bb1f18d1510deb61bae2bcb8c156f18) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #899 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/899/)
        HDFS-8151. Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. (jing9: rev 4c097e473bb1f18d1510deb61bae2bcb8c156f18)

        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #899 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/899/ ) HDFS-8151 . Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. (jing9: rev 4c097e473bb1f18d1510deb61bae2bcb8c156f18) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #166 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/166/)
        HDFS-8151. Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. (jing9: rev 4c097e473bb1f18d1510deb61bae2bcb8c156f18)

        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #166 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/166/ ) HDFS-8151 . Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. (jing9: rev 4c097e473bb1f18d1510deb61bae2bcb8c156f18) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2115 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2115/)
        HDFS-8151. Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. (jing9: rev 4c097e473bb1f18d1510deb61bae2bcb8c156f18)

        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2115 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2115/ ) HDFS-8151 . Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. (jing9: rev 4c097e473bb1f18d1510deb61bae2bcb8c156f18) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java

          People

          • Assignee:
            jingzhao Jing Zhao
            Reporter:
            ssreenivasan Sushmitha Sreenivasan
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development