Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9843

Document distcp options required for copying between encrypted locations

    Details

    • Hadoop Flags:
      Reviewed

      Description

      In TransparentEncryption.md#Distcp_considerations document section, we have "Copying_between_encrypted_and_unencrypted_locations" which requires -skipcrccheck and -update.

      These options should be documented as required for "Copying between encrypted locations" use cases as well because this involves decrypting source file and encrypting destination file with a different EDEK, resulting in different checksum at the destination. Distcp will fail at crc check if -skipcrccheck if not specified.

      This ticket is opened to document the required options for "Copying between encrypted locations" use cases when using distcp with HDFS encryption.

      1. HDFS-9843.00.patch
        3 kB
        Xiaoyu Yao
      2. HDFS-9843.01.patch
        4 kB
        Xiaoyu Yao
      3. HDFS-9843.02.patch
        4 kB
        Xiaoyu Yao

        Activity

        Hide
        xyao Xiaoyu Yao added a comment -

        Attach a initial patch.

        Show
        xyao Xiaoyu Yao added a comment - Attach a initial patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvnsite 0m 59s trunk passed
        +1 mvnsite 0m 51s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 asflicense 0m 17s Patch does not generate ASF License warnings.
        2m 29s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789103/HDFS-9843.00.patch
        JIRA Issue HDFS-9843
        Optional Tests asflicense mvnsite
        uname Linux 484c87e5e7af 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 66289a3
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14575/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvnsite 0m 59s trunk passed +1 mvnsite 0m 51s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 2m 29s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789103/HDFS-9843.00.patch JIRA Issue HDFS-9843 Optional Tests asflicense mvnsite uname Linux 484c87e5e7af 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 66289a3 modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14575/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        cnauroth Chris Nauroth added a comment -

        Hello Xiaoyu Yao. The content looks good. I'm confused about the hyperlinks for -skipcrccheck and -update though. These end up linking to TransparentEncryption.html#a-skipcrccheck and TransparentEncryption.html#a-update. There are no anchors in the page that match those, so it effectively means the browser just redisplays the same page. I also see there is an existing similar one for -px, which already exists before your patch.

        Do you know what is the intent of these hyperlinks? I checked the DistCp page for matching anchors, thinking maybe the intent is to link to a different page, but I didn't see the anchors over there either.

        Show
        cnauroth Chris Nauroth added a comment - Hello Xiaoyu Yao . The content looks good. I'm confused about the hyperlinks for -skipcrccheck and -update though. These end up linking to TransparentEncryption.html#a-skipcrccheck and TransparentEncryption.html#a-update. There are no anchors in the page that match those, so it effectively means the browser just redisplays the same page. I also see there is an existing similar one for -px, which already exists before your patch. Do you know what is the intent of these hyperlinks? I checked the DistCp page for matching anchors, thinking maybe the intent is to link to a different page, but I didn't see the anchors over there either.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Chris Nauroth for the review. Patch 01 fix the anchor to distcp command line options.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Chris Nauroth for the review. Patch 01 fix the anchor to distcp command line options.
        Hide
        cnauroth Chris Nauroth added a comment -

        Thank you, Xiaoyu Yao. I think it makes sense to clean up those links to go to the DistCp page, but they're currently not working correctly. I think you'll need to use this for the link:

        (../../hadoop-distcp/DistCp.html#Command_Line_Options)
        
        Show
        cnauroth Chris Nauroth added a comment - Thank you, Xiaoyu Yao . I think it makes sense to clean up those links to go to the DistCp page, but they're currently not working correctly. I think you'll need to use this for the link: (../../hadoop-distcp/DistCp.html#Command_Line_Options)
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Chris Nauroth! Update the patch with the fixed the link.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Chris Nauroth ! Update the patch with the fixed the link.
        Hide
        cnauroth Chris Nauroth added a comment -

        I think this patch missed out on a pre-commit run due to some Jenkins infrastructure problems yesterday. I just resubmitted a pre-commit run manually.

        Show
        cnauroth Chris Nauroth added a comment - I think this patch missed out on a pre-commit run due to some Jenkins infrastructure problems yesterday. I just resubmitted a pre-commit run manually.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvnsite 1m 4s trunk passed
        +1 mvnsite 0m 48s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        2m 38s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789334/HDFS-9843.02.patch
        JIRA Issue HDFS-9843
        Optional Tests asflicense mvnsite
        uname Linux 51bcd33a156c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 954dd57
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14602/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvnsite 1m 4s trunk passed +1 mvnsite 0m 48s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 2m 38s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789334/HDFS-9843.02.patch JIRA Issue HDFS-9843 Optional Tests asflicense mvnsite uname Linux 51bcd33a156c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 954dd57 modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14602/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        cnauroth Chris Nauroth added a comment -

        +1 for patch v02. I have committed this to trunk, branch-2 and branch-2.8. Xiaoyu Yao, thank you for the patch.

        Show
        cnauroth Chris Nauroth added a comment - +1 for patch v02. I have committed this to trunk, branch-2 and branch-2.8. Xiaoyu Yao , thank you for the patch.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thank you, Chris Nauroth for reviewing and committing the patch!

        Show
        xyao Xiaoyu Yao added a comment - Thank you, Chris Nauroth for reviewing and committing the patch!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9365 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9365/)
        HDFS-9843. Document distcp options required for copying between (cnauroth: rev dbbfc58c33fd1d2f7abae1784c2d78b7438825e2)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/TransparentEncryption.md
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9365 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9365/ ) HDFS-9843 . Document distcp options required for copying between (cnauroth: rev dbbfc58c33fd1d2f7abae1784c2d78b7438825e2) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/TransparentEncryption.md

          People

          • Assignee:
            xyao Xiaoyu Yao
            Reporter:
            xyao Xiaoyu Yao
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development