Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13827

Add reencryptEncryptedKey interface to KMS

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: kms
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      A reencryptEncryptedKey interface is added to the KMS, to re-encrypt an encrypted key with the latest version of encryption key.

      Description

      This is the KMS part. Please refer to HDFS-10899 for the design doc.

      1. HADOOP-13827.04.patch
        43 kB
        Xiao Chen
      2. HADOOP-13827.03.patch
        44 kB
        Xiao Chen
      3. HADOOP-13827.02.patch
        38 kB
        Xiao Chen
      4. HDFS-11159.01.patch
        25 kB
        Xiao Chen

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10952 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10952/)
          HADOOP-13827. Add reencryptEncryptedKey interface to KMS. (xiao: rev 79d90b810c14d5e3abab75235f587663834ce36c)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyProviderCryptoExtension.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
          • (edit) hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSAudit.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/key/TestKeyProviderCryptoExtension.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyProvider.java
          • (edit) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/EagerKeyGeneratorKeyProviderCryptoExtension.java
          • (edit) hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/LoadBalancingKMSClientProvider.java
          • (edit) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java
          • (edit) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAudit.java
          • (edit) hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm
          • (edit) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KeyAuthorizationKeyProvider.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSRESTConstants.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10952 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10952/ ) HADOOP-13827 . Add reencryptEncryptedKey interface to KMS. (xiao: rev 79d90b810c14d5e3abab75235f587663834ce36c) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyProviderCryptoExtension.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java (edit) hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSAudit.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/key/TestKeyProviderCryptoExtension.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyProvider.java (edit) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/EagerKeyGeneratorKeyProviderCryptoExtension.java (edit) hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/LoadBalancingKMSClientProvider.java (edit) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java (edit) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAudit.java (edit) hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm (edit) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KeyAuthorizationKeyProvider.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSRESTConstants.java
          Hide
          xiaochen Xiao Chen added a comment -

          Committed to trunk. Thanks Andrew Wang for the complete reviews!

          Show
          xiaochen Xiao Chen added a comment - Committed to trunk. Thanks Andrew Wang for the complete reviews!
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 43s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 25s Maven dependency ordering for branch
          +1 mvninstall 7m 23s trunk passed
          +1 compile 9m 28s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 25s trunk passed
          +1 mvneclipse 0m 36s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 1m 4s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 0m 53s the patch passed
          +1 compile 10m 18s the patch passed
          +1 javac 10m 18s the patch passed
          -0 checkstyle 0m 35s hadoop-common-project: The patch generated 2 new + 165 unchanged - 23 fixed = 167 total (was 188)
          +1 mvnsite 1m 24s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 6s the patch passed
          +1 javadoc 1m 4s the patch passed
          +1 unit 7m 23s hadoop-common in the patch passed.
          +1 unit 2m 12s hadoop-kms in the patch passed.
          +1 asflicense 0m 34s The patch does not generate ASF License warnings.
          55m 13s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13827
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842007/HADOOP-13827.04.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b2211d3f610d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 43cb167
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11204/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11204/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11204/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 43s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 25s Maven dependency ordering for branch +1 mvninstall 7m 23s trunk passed +1 compile 9m 28s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 25s trunk passed +1 mvneclipse 0m 36s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 1m 4s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 0m 53s the patch passed +1 compile 10m 18s the patch passed +1 javac 10m 18s the patch passed -0 checkstyle 0m 35s hadoop-common-project: The patch generated 2 new + 165 unchanged - 23 fixed = 167 total (was 188) +1 mvnsite 1m 24s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 1m 4s the patch passed +1 unit 7m 23s hadoop-common in the patch passed. +1 unit 2m 12s hadoop-kms in the patch passed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 55m 13s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13827 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842007/HADOOP-13827.04.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b2211d3f610d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 43cb167 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11204/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11204/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11204/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Another +1

          Show
          andrew.wang Andrew Wang added a comment - Another +1
          Hide
          xiaochen Xiao Chen added a comment -

          On second thought, since we split out HDFS-11210, let me back out those rollNewVersion changes from this commit.

          Patch 4 is identical to patch 3, except for rolling back the LBKMSCP rollNewVersion changes.

          Show
          xiaochen Xiao Chen added a comment - On second thought, since we split out HDFS-11210 , let me back out those rollNewVersion changes from this commit. Patch 4 is identical to patch 3, except for rolling back the LBKMSCP rollNewVersion changes.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Andrew for the reviews! Committing this shortly.

          Show
          xiaochen Xiao Chen added a comment - Thanks Andrew for the reviews! Committing this shortly.
          Hide
          andrew.wang Andrew Wang added a comment -

          +1 LGTM, thanks for working on this Xiao!

          Show
          andrew.wang Andrew Wang added a comment - +1 LGTM, thanks for working on this Xiao!
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 13s Maven dependency ordering for branch
          +1 mvninstall 7m 12s trunk passed
          +1 compile 9m 58s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 30s trunk passed
          +1 mvneclipse 2m 12s trunk passed
          +1 findbugs 5m 30s trunk passed
          +1 javadoc 3m 28s trunk passed
          0 mvndep 0m 30s Maven dependency ordering for patch
          +1 mvninstall 1m 55s the patch passed
          +1 compile 9m 10s the patch passed
          +1 javac 9m 10s the patch passed
          -0 checkstyle 0m 34s hadoop-common-project: The patch generated 2 new + 164 unchanged - 23 fixed = 166 total (was 187)
          +1 mvnsite 1m 23s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 6s the patch passed
          +1 javadoc 1m 4s the patch passed
          +1 unit 7m 27s hadoop-common in the patch passed.
          +1 unit 2m 13s hadoop-kms in the patch passed.
          +1 asflicense 0m 31s The patch does not generate ASF License warnings.
          64m 26s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13827
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841390/HADOOP-13827.03.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7297f80b49dc 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2d77dc7
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11179/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11179/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11179/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 7m 12s trunk passed +1 compile 9m 58s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 30s trunk passed +1 mvneclipse 2m 12s trunk passed +1 findbugs 5m 30s trunk passed +1 javadoc 3m 28s trunk passed 0 mvndep 0m 30s Maven dependency ordering for patch +1 mvninstall 1m 55s the patch passed +1 compile 9m 10s the patch passed +1 javac 9m 10s the patch passed -0 checkstyle 0m 34s hadoop-common-project: The patch generated 2 new + 164 unchanged - 23 fixed = 166 total (was 187) +1 mvnsite 1m 23s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 1m 4s the patch passed +1 unit 7m 27s hadoop-common in the patch passed. +1 unit 2m 13s hadoop-kms in the patch passed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 64m 26s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13827 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841390/HADOOP-13827.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7297f80b49dc 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2d77dc7 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11179/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11179/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11179/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 3 addressed all comments, and added docs. Didn't find any general description section, so added to the RestAPI part where the re-encrypt is introduced.

          Also crossed-out the last todo item on my list, which is drain all providers after a rollNewVersion in LBKMSCP.

          Thanks for reviewing Andrew!

          Show
          xiaochen Xiao Chen added a comment - Patch 3 addressed all comments, and added docs. Didn't find any general description section, so added to the RestAPI part where the re-encrypt is introduced. Also crossed-out the last todo item on my list, which is drain all providers after a rollNewVersion in LBKMSCP. Thanks for reviewing Andrew!
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the rev Xiao, some review comments, looks pretty close:

          • General note, we should try to use the more generic terms like EEK rather EDEK and not refer to "ezKey".
          • KeyProvider: you can use an EqualsBuilder or Guava equivalent to simplify this.
          • Thinking about it more, I guess there's no need to provide a different keyname at this point. We can compatibly add a two-parameter reencrypt method later when we need it. Sorry for doing this extra work.
          • Agree on reusing GENERATE_EEK for authorization, forgot that these are not per-op but actually classes of ops.
          Show
          andrew.wang Andrew Wang added a comment - Thanks for the rev Xiao, some review comments, looks pretty close: General note, we should try to use the more generic terms like EEK rather EDEK and not refer to "ezKey". KeyProvider: you can use an EqualsBuilder or Guava equivalent to simplify this. Thinking about it more, I guess there's no need to provide a different keyname at this point. We can compatibly add a two-parameter reencrypt method later when we need it. Sorry for doing this extra work. Agree on reusing GENERATE_EEK for authorization, forgot that these are not per-op but actually classes of ops.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 6m 49s trunk passed
          +1 compile 9m 35s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 26s trunk passed
          +1 mvneclipse 0m 37s trunk passed
          +1 findbugs 1m 54s trunk passed
          +1 javadoc 1m 6s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 0m 53s the patch passed
          +1 compile 9m 12s the patch passed
          +1 javac 9m 12s the patch passed
          -0 checkstyle 0m 34s hadoop-common-project: The patch generated 11 new + 176 unchanged - 10 fixed = 187 total (was 186)
          +1 mvnsite 1m 26s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 36s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 1m 7s the patch passed
          +1 unit 8m 28s hadoop-common in the patch passed.
          +1 unit 2m 14s hadoop-kms in the patch passed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          54m 16s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            org.apache.hadoop.crypto.key.KeyProvider$KeyVersion defines equals and uses Object.hashCode() At KeyProvider.java:Object.hashCode() At KeyProvider.java:[lines 113-129]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13827
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840818/HADOOP-13827.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6f479e85f87a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 67d9f28
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11158/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11158/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11158/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11158/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 6m 49s trunk passed +1 compile 9m 35s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 1m 6s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 53s the patch passed +1 compile 9m 12s the patch passed +1 javac 9m 12s the patch passed -0 checkstyle 0m 34s hadoop-common-project: The patch generated 11 new + 176 unchanged - 10 fixed = 187 total (was 186) +1 mvnsite 1m 26s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 36s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 7s the patch passed +1 unit 8m 28s hadoop-common in the patch passed. +1 unit 2m 14s hadoop-kms in the patch passed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 54m 16s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   org.apache.hadoop.crypto.key.KeyProvider$KeyVersion defines equals and uses Object.hashCode() At KeyProvider.java:Object.hashCode() At KeyProvider.java: [lines 113-129] Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13827 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840818/HADOOP-13827.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6f479e85f87a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 67d9f28 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11158/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11158/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11158/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11158/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the review and jira management, Andrew.

          Great idea on future proofing! Patch 2 handles that and the rest. Except:

          • ACL. Sorry my bad on wrongly using DECRYPT. But I feel REENCRYPT can share the same ACL as GENERATE, since they behave really similarly - ask for a (re)generated EDEK. KMSACLs#Type and KMS#KMSOp are not 1-1 mapping, so in this patch I used generate acl for reencrypt op. Please let me know if you feel otherwise.
          • Doc update will come in later, after things stabilize a bit. Added a line in the doc so it's not forgotten in later revs.
          Show
          xiaochen Xiao Chen added a comment - Thanks for the review and jira management, Andrew. Great idea on future proofing! Patch 2 handles that and the rest. Except: ACL. Sorry my bad on wrongly using DECRYPT. But I feel REENCRYPT can share the same ACL as GENERATE, since they behave really similarly - ask for a (re)generated EDEK. KMSACLs#Type and KMS#KMSOp are not 1-1 mapping, so in this patch I used generate acl for reencrypt op. Please let me know if you feel otherwise. Doc update will come in later, after things stabilize a bit. Added a line in the doc so it's not forgotten in later revs.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 21s Maven dependency ordering for branch
          +1 mvninstall 7m 30s trunk passed
          +1 compile 11m 11s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 31s trunk passed
          +1 mvneclipse 0m 34s trunk passed
          +1 findbugs 2m 7s trunk passed
          +1 javadoc 1m 6s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 0m 54s the patch passed
          +1 compile 9m 32s the patch passed
          +1 javac 9m 32s the patch passed
          -0 checkstyle 0m 34s hadoop-common-project: The patch generated 1 new + 39 unchanged - 2 fixed = 40 total (was 41)
          +1 mvnsite 1m 26s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 17s the patch passed
          +1 javadoc 1m 7s the patch passed
          +1 unit 8m 42s hadoop-common in the patch passed.
          +1 unit 2m 11s hadoop-kms in the patch passed.
          +1 asflicense 0m 31s The patch does not generate ASF License warnings.
          57m 38s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13827
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839664/HDFS-11159.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 268fd3168b15 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2bf9a15
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11117/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11117/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11117/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 21s Maven dependency ordering for branch +1 mvninstall 7m 30s trunk passed +1 compile 11m 11s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 31s trunk passed +1 mvneclipse 0m 34s trunk passed +1 findbugs 2m 7s trunk passed +1 javadoc 1m 6s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 54s the patch passed +1 compile 9m 32s the patch passed +1 javac 9m 32s the patch passed -0 checkstyle 0m 34s hadoop-common-project: The patch generated 1 new + 39 unchanged - 2 fixed = 40 total (was 41) +1 mvnsite 1m 26s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 17s the patch passed +1 javadoc 1m 7s the patch passed +1 unit 8m 42s hadoop-common in the patch passed. +1 unit 2m 11s hadoop-kms in the patch passed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 57m 38s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13827 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839664/HDFS-11159.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 268fd3168b15 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2bf9a15 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11117/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11117/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11117/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for splitting this out Xiao. Some review comments:

          • It's be more future proof if the reencryption API also took a keyName. This way we could rotate to a new key entirely, or rename between encryption zones.
          • Regarding authorization, reencrypt right now reuses the DECRYPT_EEK ACL. We separated out the GENERATE_EEK and DECRYPT_EEK so that the namenode doesn't need DECRYPT_EEK. Also, the other ops all have per-op ACLs, so this should probably be per-op as well.
          • Related, I'd like to see some authorization-related tests.
          • KMSAudit whitelists DECRYPT_EEK and GENERATE_EEK, should probably add REENCRYPT_EEK as well. Audit test update would be good too.
          • A doc update to explain this new op and what additionally needs to be configured would be good.
          • We need KMS-level tests as well, looks like the added tests only cover the KPCE implementation.
          Show
          andrew.wang Andrew Wang added a comment - Thanks for splitting this out Xiao. Some review comments: It's be more future proof if the reencryption API also took a keyName. This way we could rotate to a new key entirely, or rename between encryption zones. Regarding authorization, reencrypt right now reuses the DECRYPT_EEK ACL. We separated out the GENERATE_EEK and DECRYPT_EEK so that the namenode doesn't need DECRYPT_EEK. Also, the other ops all have per-op ACLs, so this should probably be per-op as well. Related, I'd like to see some authorization-related tests. KMSAudit whitelists DECRYPT_EEK and GENERATE_EEK, should probably add REENCRYPT_EEK as well. Audit test update would be good too. A doc update to explain this new op and what additionally needs to be configured would be good. We need KMS-level tests as well, looks like the added tests only cover the KPCE implementation.
          Hide
          andrew.wang Andrew Wang added a comment -

          I went ahead and moved this to HADOOP and linked it to HDFS-10899, the relationship should still be clear to watchers.

          Show
          andrew.wang Andrew Wang added a comment - I went ahead and moved this to HADOOP and linked it to HDFS-10899 , the relationship should still be clear to watchers.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 21s Maven dependency ordering for branch
          +1 mvninstall 7m 31s trunk passed
          +1 compile 10m 4s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 1m 31s trunk passed
          +1 mvneclipse 0m 35s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 1m 4s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 0m 57s the patch passed
          +1 compile 9m 44s the patch passed
          +1 javac 9m 44s the patch passed
          -0 checkstyle 0m 33s hadoop-common-project: The patch generated 1 new + 42 unchanged - 2 fixed = 43 total (was 44)
          +1 mvnsite 1m 28s the patch passed
          +1 mvneclipse 0m 35s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 10s the patch passed
          +1 javadoc 1m 5s the patch passed
          +1 unit 8m 4s hadoop-common in the patch passed.
          +1 unit 2m 13s hadoop-kms in the patch passed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          52m 22s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11159
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839664/HDFS-11159.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 17e44b253219 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7584fbf
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17614/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17614/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17614/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 21s Maven dependency ordering for branch +1 mvninstall 7m 31s trunk passed +1 compile 10m 4s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 1m 31s trunk passed +1 mvneclipse 0m 35s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 1m 4s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 57s the patch passed +1 compile 9m 44s the patch passed +1 javac 9m 44s the patch passed -0 checkstyle 0m 33s hadoop-common-project: The patch generated 1 new + 42 unchanged - 2 fixed = 43 total (was 44) +1 mvnsite 1m 28s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 10s the patch passed +1 javadoc 1m 5s the patch passed +1 unit 8m 4s hadoop-common in the patch passed. +1 unit 2m 13s hadoop-kms in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 52m 22s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11159 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839664/HDFS-11159.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 17e44b253219 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7584fbf Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17614/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17614/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17614/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 1 attached. Ideally this should be a hadoop jira but no such option as a subtask of HDFS-10899.

          Show
          xiaochen Xiao Chen added a comment - Patch 1 attached. Ideally this should be a hadoop jira but no such option as a subtask of HDFS-10899 .

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              xiaochen Xiao Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development