Hadoop Common
  1. Hadoop Common
  2. HADOOP-10433

Key Management Server based on KeyProvider API

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 2.6.0
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      (from HDFS-6134 proposal)

      Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality).

      Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177.

      Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation.

      The client-server protocol will be secure:

      • Kerberos HTTP SPNEGO (authentication)
      • HTTPS for transport (confidentiality and integrity)
      • Hadoop ACLs (authorization)

      The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used.

      Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool

      There are minor changes that must be done in Hadoop KeyProvider functionality:

      The KeyProvider contract, and the existing implementations, must be thread-safe

      KeyProvider API should have an API to generate the key material internally
      JavaKeyStoreProvider should use, if present, a password provided via configuration

      KeyProvider Option and Metadata should include a label (for easier cross-referencing)

      To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy.

      Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available.

      1. KMS-doc.pdf
        237 kB
        Alejandro Abdelnur
      2. HadoopKMSDocsv2.pdf
        527 kB
        Alejandro Abdelnur
      3. HADOOP-10433.patch
        183 kB
        Alejandro Abdelnur
      4. HADOOP-10433.patch
        183 kB
        Alejandro Abdelnur
      5. HADOOP-10433.patch
        201 kB
        Alejandro Abdelnur
      6. HADOOP-10433.patch
        201 kB
        Alejandro Abdelnur
      7. HADOOP-10433.patch
        201 kB
        Alejandro Abdelnur
      8. HADOOP-10433.patch
        202 kB
        Alejandro Abdelnur
      9. HADOOP-10433.patch
        203 kB
        Alejandro Abdelnur
      10. HADOOP-10433.patch
        203 kB
        Alejandro Abdelnur
      11. HADOOP-10433.patch
        191 kB
        Alejandro Abdelnur
      12. HADOOP-10433.patch
        191 kB
        Alejandro Abdelnur
      13. HADOOP-10433.patch
        191 kB
        Alejandro Abdelnur
      14. HADOOP-10433.patch
        192 kB
        Alejandro Abdelnur

        Issue Links

          Activity

          Hide
          mohamed Abo el haggag hassany added a comment -

          sorry guys can any one give me a document describe the process of the authentication

          Show
          mohamed Abo el haggag hassany added a comment - sorry guys can any one give me a document describe the process of the authentication
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1777 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1777/)
          HADOOP-10433. Key Management Server based on KeyProvider API. (tucu) (tucu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1592637)

          • /hadoop/common/trunk/.gitignore
          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-kms-dist.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSRESTConstants.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-acls.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSACLs.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAudit.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSCacheKeyProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSConfiguration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSMDCFilter.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSServerJSONUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF/web.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/index.html
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/logging.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/server.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ssl-server.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF/web.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/apt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/apt/index.apt.vm
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources/css
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources/css/site.css
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSACLs.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSCacheKeyProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/resources
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/resources/log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/pom.xml
          • /hadoop/common/trunk/hadoop-dist/pom.xml
          • /hadoop/common/trunk/hadoop-project/pom.xml
          • /hadoop/common/trunk/hadoop-project/src/site/site.xml
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1777 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1777/ ) HADOOP-10433 . Key Management Server based on KeyProvider API. (tucu) (tucu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1592637 ) /hadoop/common/trunk/.gitignore /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-kms-dist.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSRESTConstants.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory /hadoop/common/trunk/hadoop-common-project/hadoop-kms /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-acls.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSACLs.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAudit.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSCacheKeyProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSConfiguration.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSMDCFilter.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSServerJSONUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF/web.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/index.html /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/logging.properties /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/server.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ssl-server.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF/web.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/apt /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/apt/index.apt.vm /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources/css /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources/css/site.css /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSACLs.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSCacheKeyProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/resources /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/resources/log4j.properties /hadoop/common/trunk/hadoop-common-project/pom.xml /hadoop/common/trunk/hadoop-dist/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml /hadoop/common/trunk/hadoop-project/src/site/site.xml
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1751 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1751/)
          HADOOP-10433. Key Management Server based on KeyProvider API. (tucu) (tucu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1592637)

          • /hadoop/common/trunk/.gitignore
          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-kms-dist.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSRESTConstants.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-acls.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSACLs.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAudit.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSCacheKeyProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSConfiguration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSMDCFilter.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSServerJSONUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF/web.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/index.html
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/logging.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/server.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ssl-server.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF/web.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/apt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/apt/index.apt.vm
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources/css
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources/css/site.css
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSACLs.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSCacheKeyProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/resources
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/resources/log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/pom.xml
          • /hadoop/common/trunk/hadoop-dist/pom.xml
          • /hadoop/common/trunk/hadoop-project/pom.xml
          • /hadoop/common/trunk/hadoop-project/src/site/site.xml
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1751 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1751/ ) HADOOP-10433 . Key Management Server based on KeyProvider API. (tucu) (tucu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1592637 ) /hadoop/common/trunk/.gitignore /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-kms-dist.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSRESTConstants.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory /hadoop/common/trunk/hadoop-common-project/hadoop-kms /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-acls.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSACLs.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAudit.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSCacheKeyProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSConfiguration.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSMDCFilter.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSServerJSONUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF/web.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/index.html /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/logging.properties /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/server.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ssl-server.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF/web.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/apt /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/apt/index.apt.vm /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources/css /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources/css/site.css /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSACLs.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSCacheKeyProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/resources /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/resources/log4j.properties /hadoop/common/trunk/hadoop-common-project/pom.xml /hadoop/common/trunk/hadoop-dist/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml /hadoop/common/trunk/hadoop-project/src/site/site.xml
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #559 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/559/)
          HADOOP-10433. Key Management Server based on KeyProvider API. (tucu) (tucu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1592637)

          • /hadoop/common/trunk/.gitignore
          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-kms-dist.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSRESTConstants.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-acls.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSACLs.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAudit.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSCacheKeyProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSConfiguration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSMDCFilter.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSServerJSONUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF/web.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/index.html
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/logging.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/server.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ssl-server.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF/web.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/apt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/apt/index.apt.vm
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources/css
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources/css/site.css
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSACLs.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSCacheKeyProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/resources
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/resources/log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/pom.xml
          • /hadoop/common/trunk/hadoop-dist/pom.xml
          • /hadoop/common/trunk/hadoop-project/pom.xml
          • /hadoop/common/trunk/hadoop-project/src/site/site.xml
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #559 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/559/ ) HADOOP-10433 . Key Management Server based on KeyProvider API. (tucu) (tucu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1592637 ) /hadoop/common/trunk/.gitignore /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-kms-dist.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSRESTConstants.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory /hadoop/common/trunk/hadoop-common-project/hadoop-kms /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-acls.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSACLs.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAudit.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSCacheKeyProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSConfiguration.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSMDCFilter.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSServerJSONUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF/web.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/index.html /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/logging.properties /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/server.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ssl-server.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF/web.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/apt /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/apt/index.apt.vm /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources/css /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources/css/site.css /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSACLs.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSCacheKeyProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/resources /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/resources/log4j.properties /hadoop/common/trunk/hadoop-common-project/pom.xml /hadoop/common/trunk/hadoop-dist/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml /hadoop/common/trunk/hadoop-project/src/site/site.xml
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5593 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5593/)
          HADOOP-10433. Key Management Server based on KeyProvider API. (tucu) (tucu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1592637)

          • /hadoop/common/trunk/.gitignore
          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-kms-dist.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSRESTConstants.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-acls.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSACLs.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAudit.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSCacheKeyProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSConfiguration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSMDCFilter.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSServerJSONUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF/web.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/index.html
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/logging.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/server.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ssl-server.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF/web.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/apt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/apt/index.apt.vm
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources/css
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources/css/site.css
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSACLs.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSCacheKeyProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/resources
          • /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/resources/log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/pom.xml
          • /hadoop/common/trunk/hadoop-dist/pom.xml
          • /hadoop/common/trunk/hadoop-project/pom.xml
          • /hadoop/common/trunk/hadoop-project/src/site/site.xml
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5593 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5593/ ) HADOOP-10433 . Key Management Server based on KeyProvider API. (tucu) (tucu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1592637 ) /hadoop/common/trunk/.gitignore /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-kms-dist.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSRESTConstants.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory /hadoop/common/trunk/hadoop-common-project/hadoop-kms /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-acls.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSACLs.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAudit.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSCacheKeyProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSConfiguration.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSMDCFilter.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSServerJSONUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF/web.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/index.html /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/logging.properties /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/server.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ssl-server.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF/web.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/apt /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/apt/index.apt.vm /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources/css /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/resources/css/site.css /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/site/site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSACLs.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMSCacheKeyProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/resources /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/test/resources/log4j.properties /hadoop/common/trunk/hadoop-common-project/pom.xml /hadoop/common/trunk/hadoop-dist/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml /hadoop/common/trunk/hadoop-project/src/site/site.xml
          Hide
          Larry McCay added a comment -

          Thanks for all the heavy lifting here, Alejandro Abdelnur!

          Show
          Larry McCay added a comment - Thanks for all the heavy lifting here, Alejandro Abdelnur !
          Hide
          Alejandro Abdelnur added a comment -

          Committed to trunk. Thanks to all for the reviews.

          Show
          Alejandro Abdelnur added a comment - Committed to trunk. Thanks to all for the reviews.
          Hide
          Alejandro Abdelnur added a comment -

          planning to commit on monday

          Show
          Alejandro Abdelnur added a comment - planning to commit on monday
          Hide
          Alejandro Abdelnur added a comment -

          Owen? are we good as the initial commit, and we follow up with other JIRAs improvements?

          Show
          Alejandro Abdelnur added a comment - Owen? are we good as the initial commit, and we follow up with other JIRAs improvements?
          Hide
          Andrew Wang added a comment -

          LGTM, +1 again

          Show
          Andrew Wang added a comment - LGTM, +1 again
          Hide
          Vinay Shukla added a comment -

          Thanks for the sample Audit record. I have created a JIRA for normalizing audit records. https://issues.apache.org/jira/browse/HADOOP-10569

          Show
          Vinay Shukla added a comment - Thanks for the sample Audit record. I have created a JIRA for normalizing audit records. https://issues.apache.org/jira/browse/HADOOP-10569
          Hide
          Alejandro Abdelnur added a comment -

          Test failure seems unrelated.

          Owen, Andrew, are we good to go now?

          Vinay, a sample of the audit log follows. Regarding normalizing audit logs, that would be nice, mind opening a JIRA for it (it is out of scope from this JIRA).

          2014-05-02 10:14:12,326 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65499/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:12,994 Status:OK User:client@EXAMPLE.COM Op:CREATE_KEY Name:ck0UserProvidedMaterial:false Description:null
          2014-05-02 10:14:13,027 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65499/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:13,076 Status:OK User:client/host@EXAMPLE.COM Op:CREATE_KEY Name:ck1UserProvidedMaterial:false Description:null
          2014-05-02 10:14:13,557 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:13,597 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:CREATE_KEY Name:k
          2014-05-02 10:14:13,597 Status:ERROR User:client@EXAMPLE.COM Method:POST URL:http://localhost:65509/kms/v1/keys Exception:'User:client@EXAMPLE.COM not allowed to do 'CREATE_KEY' on 'k''
          2014-05-02 10:14:13,606 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:13,645 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:CREATE_KEY Name:k
          2014-05-02 10:14:13,646 Status:ERROR User:client@EXAMPLE.COM Method:POST URL:http://localhost:65509/kms/v1/keys Exception:'User:client@EXAMPLE.COM not allowed to do 'CREATE_KEY' on 'k''
          2014-05-02 10:14:13,653 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/key/k?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:13,692 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:ROLL_NEW_VERSION Name:k
          2014-05-02 10:14:13,692 Status:ERROR User:client@EXAMPLE.COM Method:POST URL:http://localhost:65509/kms/v1/key/k Exception:'User:client@EXAMPLE.COM not allowed to do 'ROLL_NEW_VERSION' on 'k''
          2014-05-02 10:14:13,699 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/key/k?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:13,739 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:ROLL_NEW_VERSION Name:k
          2014-05-02 10:14:13,739 Status:ERROR User:client@EXAMPLE.COM Method:POST URL:http://localhost:65509/kms/v1/key/k Exception:'User:client@EXAMPLE.COM not allowed to do 'ROLL_NEW_VERSION' on 'k''
          2014-05-02 10:14:13,746 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/keys/names?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:13,781 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:GET_KEYS Name:*
          2014-05-02 10:14:13,781 Status:ERROR User:client@EXAMPLE.COM Method:GET URL:http://localhost:65509/kms/v1/keys/names Exception:'User:client@EXAMPLE.COM not allowed to do 'GET_KEYS' on '*''
          2014-05-02 10:14:13,791 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/keys/metadata?key=k&user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:13,825 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:GET_KEYS_METADATA Name:k
          2014-05-02 10:14:13,826 Status:ERROR User:client@EXAMPLE.COM Method:GET URL:http://localhost:65509/kms/v1/keys/metadata?key=k Exception:'User:client@EXAMPLE.COM not allowed to do 'GET_KEYS_METADATA' on 'k''
          2014-05-02 10:14:13,836 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/keyversion/k%400?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:13,871 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:GET_KEY_VERSION Name:k@0
          2014-05-02 10:14:13,871 Status:ERROR User:client@EXAMPLE.COM Method:GET URL:http://localhost:65509/kms/v1/keyversion/k%400 Exception:'User:client@EXAMPLE.COM not allowed to do 'GET_KEY_VERSION' on 'k@0''
          2014-05-02 10:14:13,878 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/key/k/_currentversion?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:13,912 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:GET_CURRENT_KEY Name:k
          2014-05-02 10:14:13,912 Status:ERROR User:client@EXAMPLE.COM Method:GET URL:http://localhost:65509/kms/v1/key/k/_currentversion Exception:'User:client@EXAMPLE.COM not allowed to do 'GET_CURRENT_KEY' on 'k''
          2014-05-02 10:14:13,919 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/key/k/_metadata?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:13,952 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:GET_METADATA Name:k
          2014-05-02 10:14:13,952 Status:ERROR User:client@EXAMPLE.COM Method:GET URL:http://localhost:65509/kms/v1/key/k/_metadata Exception:'User:client@EXAMPLE.COM not allowed to do 'GET_METADATA' on 'k''
          2014-05-02 10:14:13,959 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/key/k/_versions?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:13,991 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:GET_KEY_VERSIONS Name:k
          2014-05-02 10:14:13,992 Status:ERROR User:client@EXAMPLE.COM Method:GET URL:http://localhost:65509/kms/v1/key/k/_versions Exception:'User:client@EXAMPLE.COM not allowed to do 'GET_KEY_VERSIONS' on 'k''
          2014-05-02 10:14:14,010 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:14,054 Status:OK User:CREATE@EXAMPLE.COM Op:CREATE_KEY Name:k0UserProvidedMaterial:false Description:null
          2014-05-02 10:14:14,081 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/key/k0?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:14,121 Status:OK User:DELETE@EXAMPLE.COM Op:DELETE_KEY Name:k0
          2014-05-02 10:14:14,135 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:14,184 Status:OK User:SET_KEY_MATERIAL@EXAMPLE.COM Op:CREATE_KEY Name:k1UserProvidedMaterial:true Description:null
          2014-05-02 10:14:14,210 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/key/k1?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:14,253 Status:OK User:ROLLOVER@EXAMPLE.COM Op:ROLL_NEW_VERSION Name:k1UserProvidedMaterial:false NewVersion:k1@1
          2014-05-02 10:14:14,280 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/key/k1?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:14,323 Status:OK User:SET_KEY_MATERIAL@EXAMPLE.COM Op:ROLL_NEW_VERSION Name:k1UserProvidedMaterial:true NewVersion:k1@2
          2014-05-02 10:14:14,352 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/keyversion/k1%400?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:14,382 Status:OK User:GET@EXAMPLE.COM Op:GET_KEY_VERSION Name:k1@0
          2014-05-02 10:14:14,387 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/key/k1/_currentversion?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:14,408 Status:OK User:GET@EXAMPLE.COM Op:GET_CURRENT_KEY Name:k1
          2014-05-02 10:14:14,422 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/keys/names?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:14,441 Status:OK User:GET_KEYS@EXAMPLE.COM Op:GET_KEYS Name:*
          2014-05-02 10:14:14,461 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/key/k1/_metadata?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:14,480 Status:OK User:GET_METADATA@EXAMPLE.COM Op:GET_METADATA Name:k1
          2014-05-02 10:14:14,486 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/keys/metadata?key=k1&user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:14,503 Status:OK User:GET_METADATA@EXAMPLE.COM Op:GET_KEYS_METADATA Name:k1
          2014-05-02 10:14:14,586 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:14,618 Status:UNAUTHORIZED User:CREATE@EXAMPLE.COM Op:CREATE_KEY Name:k2
          2014-05-02 10:14:14,619 Status:ERROR User:CREATE@EXAMPLE.COM Method:POST URL:http://localhost:65509/kms/v1/keys Exception:'User:CREATE@EXAMPLE.COM not allowed to do 'CREATE_KEY' on 'k2''
          2014-05-02 10:14:14,962 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:49200/kms/v1/keys/names?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:14,983 Status:OK User:client@EXAMPLE.COM Op:GET_KEYS Name:*
          2014-05-02 10:14:14,996 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:49200/kms/v1/keys/names?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:15,017 Status:OK User:client/host@EXAMPLE.COM Op:GET_KEYS Name:*
          2014-05-02 10:14:15,417 Status:OK User:tucu Op:GET_KEYS Name:*
          2014-05-02 10:14:15,979 Status:OK User:tucu Op:GET_KEYS Name:*
          2014-05-02 10:14:16,388 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:https://localhost:49216/kms/v1/keys/names?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:16,420 Status:OK User:client@EXAMPLE.COM Op:GET_KEYS Name:*
          2014-05-02 10:14:16,437 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:https://localhost:49216/kms/v1/keys/names?user.name=tucu ErrorMsg:'Authentication required'
          2014-05-02 10:14:16,466 Status:OK User:client/host@EXAMPLE.COM Op:GET_KEYS Name:*
          2014-05-02 10:14:16,774 Status:OK User:tucu Op:GET_KEYS Name:*
          2014-05-02 10:14:16,812 Status:OK User:tucu Op:CREATE_KEY Name:k1UserProvidedMaterial:false Description:l1
          2014-05-02 10:14:16,824 Status:OK User:tucu Op:GET_KEY_VERSION Name:k1@0
          2014-05-02 10:14:16,834 Status:OK User:tucu Op:GET_CURRENT_KEY Name:k1
          2014-05-02 10:14:16,842 Status:OK User:tucu Op:GET_METADATA Name:k1
          2014-05-02 10:14:16,852 Status:OK User:tucu Op:GET_KEY_VERSIONS Name:k1
          2014-05-02 10:14:16,884 Status:OK User:tucu Op:ROLL_NEW_VERSION Name:k1UserProvidedMaterial:false NewVersion:k1@1
          2014-05-02 10:14:16,895 Status:OK User:tucu Op:GET_KEY_VERSION Name:k1@1
          2014-05-02 10:14:16,903 Status:OK User:tucu Op:GET_CURRENT_KEY Name:k1
          2014-05-02 10:14:16,913 Status:OK User:tucu Op:GET_KEY_VERSIONS Name:k1
          2014-05-02 10:14:16,922 Status:OK User:tucu Op:GET_METADATA Name:k1
          2014-05-02 10:14:16,930 Status:OK User:tucu Op:GET_KEYS Name:*
          2014-05-02 10:14:16,937 Status:OK User:tucu Op:GET_KEYS_METADATA Name:k1
          2014-05-02 10:14:16,962 Status:OK User:tucu Op:DELETE_KEY Name:k1
          2014-05-02 10:14:16,969 Status:OK User:tucu Op:GET_KEY_VERSION Name:k1
          2014-05-02 10:14:16,978 Status:OK User:tucu Op:GET_KEY_VERSIONS Name:k1
          2014-05-02 10:14:16,985 Status:OK User:tucu Op:GET_METADATA Name:k1
          2014-05-02 10:14:16,991 Status:OK User:tucu Op:GET_KEYS Name:*
          
          Show
          Alejandro Abdelnur added a comment - Test failure seems unrelated. Owen, Andrew, are we good to go now? Vinay, a sample of the audit log follows. Regarding normalizing audit logs, that would be nice, mind opening a JIRA for it (it is out of scope from this JIRA). 2014-05-02 10:14:12,326 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65499/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:12,994 Status:OK User:client@EXAMPLE.COM Op:CREATE_KEY Name:ck0UserProvidedMaterial: false Description: null 2014-05-02 10:14:13,027 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65499/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,076 Status:OK User:client/host@EXAMPLE.COM Op:CREATE_KEY Name:ck1UserProvidedMaterial: false Description: null 2014-05-02 10:14:13,557 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,597 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:CREATE_KEY Name:k 2014-05-02 10:14:13,597 Status:ERROR User:client@EXAMPLE.COM Method:POST URL:http: //localhost:65509/kms/v1/keys Exception:'User:client@EXAMPLE.COM not allowed to do 'CREATE_KEY' on 'k'' 2014-05-02 10:14:13,606 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,645 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:CREATE_KEY Name:k 2014-05-02 10:14:13,646 Status:ERROR User:client@EXAMPLE.COM Method:POST URL:http: //localhost:65509/kms/v1/keys Exception:'User:client@EXAMPLE.COM not allowed to do 'CREATE_KEY' on 'k'' 2014-05-02 10:14:13,653 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/key/k?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,692 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:ROLL_NEW_VERSION Name:k 2014-05-02 10:14:13,692 Status:ERROR User:client@EXAMPLE.COM Method:POST URL:http: //localhost:65509/kms/v1/key/k Exception:'User:client@EXAMPLE.COM not allowed to do 'ROLL_NEW_VERSION' on 'k'' 2014-05-02 10:14:13,699 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/key/k?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,739 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:ROLL_NEW_VERSION Name:k 2014-05-02 10:14:13,739 Status:ERROR User:client@EXAMPLE.COM Method:POST URL:http: //localhost:65509/kms/v1/key/k Exception:'User:client@EXAMPLE.COM not allowed to do 'ROLL_NEW_VERSION' on 'k'' 2014-05-02 10:14:13,746 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/keys/names?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,781 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:GET_KEYS Name:* 2014-05-02 10:14:13,781 Status:ERROR User:client@EXAMPLE.COM Method:GET URL:http: //localhost:65509/kms/v1/keys/names Exception:'User:client@EXAMPLE.COM not allowed to do 'GET_KEYS' on '*'' 2014-05-02 10:14:13,791 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/keys/metadata?key=k&user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,825 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:GET_KEYS_METADATA Name:k 2014-05-02 10:14:13,826 Status:ERROR User:client@EXAMPLE.COM Method:GET URL:http: //localhost:65509/kms/v1/keys/metadata?key=k Exception:'User:client@EXAMPLE.COM not allowed to do 'GET_KEYS_METADATA' on 'k'' 2014-05-02 10:14:13,836 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/keyversion/k%400?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,871 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:GET_KEY_VERSION Name:k@0 2014-05-02 10:14:13,871 Status:ERROR User:client@EXAMPLE.COM Method:GET URL:http: //localhost:65509/kms/v1/keyversion/k%400 Exception:'User:client@EXAMPLE.COM not allowed to do 'GET_KEY_VERSION' on 'k@0'' 2014-05-02 10:14:13,878 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/key/k/_currentversion?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,912 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:GET_CURRENT_KEY Name:k 2014-05-02 10:14:13,912 Status:ERROR User:client@EXAMPLE.COM Method:GET URL:http: //localhost:65509/kms/v1/key/k/_currentversion Exception:'User:client@EXAMPLE.COM not allowed to do 'GET_CURRENT_KEY' on 'k'' 2014-05-02 10:14:13,919 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/key/k/_metadata?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,952 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:GET_METADATA Name:k 2014-05-02 10:14:13,952 Status:ERROR User:client@EXAMPLE.COM Method:GET URL:http: //localhost:65509/kms/v1/key/k/_metadata Exception:'User:client@EXAMPLE.COM not allowed to do 'GET_METADATA' on 'k'' 2014-05-02 10:14:13,959 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/key/k/_versions?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,991 Status:UNAUTHORIZED User:client@EXAMPLE.COM Op:GET_KEY_VERSIONS Name:k 2014-05-02 10:14:13,992 Status:ERROR User:client@EXAMPLE.COM Method:GET URL:http: //localhost:65509/kms/v1/key/k/_versions Exception:'User:client@EXAMPLE.COM not allowed to do 'GET_KEY_VERSIONS' on 'k'' 2014-05-02 10:14:14,010 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:14,054 Status:OK User:CREATE@EXAMPLE.COM Op:CREATE_KEY Name:k0UserProvidedMaterial: false Description: null 2014-05-02 10:14:14,081 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/key/k0?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:14,121 Status:OK User:DELETE@EXAMPLE.COM Op:DELETE_KEY Name:k0 2014-05-02 10:14:14,135 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:14,184 Status:OK User:SET_KEY_MATERIAL@EXAMPLE.COM Op:CREATE_KEY Name:k1UserProvidedMaterial: true Description: null 2014-05-02 10:14:14,210 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/key/k1?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:14,253 Status:OK User:ROLLOVER@EXAMPLE.COM Op:ROLL_NEW_VERSION Name:k1UserProvidedMaterial: false NewVersion:k1@1 2014-05-02 10:14:14,280 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/key/k1?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:14,323 Status:OK User:SET_KEY_MATERIAL@EXAMPLE.COM Op:ROLL_NEW_VERSION Name:k1UserProvidedMaterial: true NewVersion:k1@2 2014-05-02 10:14:14,352 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/keyversion/k1%400?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:14,382 Status:OK User:GET@EXAMPLE.COM Op:GET_KEY_VERSION Name:k1@0 2014-05-02 10:14:14,387 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/key/k1/_currentversion?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:14,408 Status:OK User:GET@EXAMPLE.COM Op:GET_CURRENT_KEY Name:k1 2014-05-02 10:14:14,422 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/keys/names?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:14,441 Status:OK User:GET_KEYS@EXAMPLE.COM Op:GET_KEYS Name:* 2014-05-02 10:14:14,461 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/key/k1/_metadata?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:14,480 Status:OK User:GET_METADATA@EXAMPLE.COM Op:GET_METADATA Name:k1 2014-05-02 10:14:14,486 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/keys/metadata?key=k1&user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:14,503 Status:OK User:GET_METADATA@EXAMPLE.COM Op:GET_KEYS_METADATA Name:k1 2014-05-02 10:14:14,586 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:65509/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:14,618 Status:UNAUTHORIZED User:CREATE@EXAMPLE.COM Op:CREATE_KEY Name:k2 2014-05-02 10:14:14,619 Status:ERROR User:CREATE@EXAMPLE.COM Method:POST URL:http: //localhost:65509/kms/v1/keys Exception:'User:CREATE@EXAMPLE.COM not allowed to do 'CREATE_KEY' on 'k2'' 2014-05-02 10:14:14,962 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:49200/kms/v1/keys/names?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:14,983 Status:OK User:client@EXAMPLE.COM Op:GET_KEYS Name:* 2014-05-02 10:14:14,996 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http: //localhost:49200/kms/v1/keys/names?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:15,017 Status:OK User:client/host@EXAMPLE.COM Op:GET_KEYS Name:* 2014-05-02 10:14:15,417 Status:OK User:tucu Op:GET_KEYS Name:* 2014-05-02 10:14:15,979 Status:OK User:tucu Op:GET_KEYS Name:* 2014-05-02 10:14:16,388 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:https: //localhost:49216/kms/v1/keys/names?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:16,420 Status:OK User:client@EXAMPLE.COM Op:GET_KEYS Name:* 2014-05-02 10:14:16,437 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:https: //localhost:49216/kms/v1/keys/names?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:16,466 Status:OK User:client/host@EXAMPLE.COM Op:GET_KEYS Name:* 2014-05-02 10:14:16,774 Status:OK User:tucu Op:GET_KEYS Name:* 2014-05-02 10:14:16,812 Status:OK User:tucu Op:CREATE_KEY Name:k1UserProvidedMaterial: false Description:l1 2014-05-02 10:14:16,824 Status:OK User:tucu Op:GET_KEY_VERSION Name:k1@0 2014-05-02 10:14:16,834 Status:OK User:tucu Op:GET_CURRENT_KEY Name:k1 2014-05-02 10:14:16,842 Status:OK User:tucu Op:GET_METADATA Name:k1 2014-05-02 10:14:16,852 Status:OK User:tucu Op:GET_KEY_VERSIONS Name:k1 2014-05-02 10:14:16,884 Status:OK User:tucu Op:ROLL_NEW_VERSION Name:k1UserProvidedMaterial: false NewVersion:k1@1 2014-05-02 10:14:16,895 Status:OK User:tucu Op:GET_KEY_VERSION Name:k1@1 2014-05-02 10:14:16,903 Status:OK User:tucu Op:GET_CURRENT_KEY Name:k1 2014-05-02 10:14:16,913 Status:OK User:tucu Op:GET_KEY_VERSIONS Name:k1 2014-05-02 10:14:16,922 Status:OK User:tucu Op:GET_METADATA Name:k1 2014-05-02 10:14:16,930 Status:OK User:tucu Op:GET_KEYS Name:* 2014-05-02 10:14:16,937 Status:OK User:tucu Op:GET_KEYS_METADATA Name:k1 2014-05-02 10:14:16,962 Status:OK User:tucu Op:DELETE_KEY Name:k1 2014-05-02 10:14:16,969 Status:OK User:tucu Op:GET_KEY_VERSION Name:k1 2014-05-02 10:14:16,978 Status:OK User:tucu Op:GET_KEY_VERSIONS Name:k1 2014-05-02 10:14:16,985 Status:OK User:tucu Op:GET_METADATA Name:k1 2014-05-02 10:14:16,991 Status:OK User:tucu Op:GET_KEYS Name:*
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642989/HADOOP-10433.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist:

          org.apache.hadoop.ha.TestZKFailoverControllerStress

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3885//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3885//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642989/HADOOP-10433.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist: org.apache.hadoop.ha.TestZKFailoverControllerStress +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3885//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3885//console This message is automatically generated.
          Hide
          Vinay Shukla added a comment -

          Is there a sample of the audit log record? What field are audited? Would it be useful to have a common audit format across Hadoop?

          Show
          Vinay Shukla added a comment - Is there a sample of the audit log record? What field are audited? Would it be useful to have a common audit format across Hadoop?
          Hide
          Alejandro Abdelnur added a comment -

          new patch addressing Owen's suggestions.

          Show
          Alejandro Abdelnur added a comment - new patch addressing Owen's suggestions.
          Hide
          Alejandro Abdelnur added a comment -

          Vinay, there is a kms-audit.log file that logs all requests, both successful and failed.

          Show
          Alejandro Abdelnur added a comment - Vinay, there is a kms-audit.log file that logs all requests, both successful and failed.
          Hide
          Vinay Shukla added a comment -

          Alejandro Abdelnur What is the audit story about the Key? Do we record, who did various key operations? I couldn't find it in the attached PDFs.

          Show
          Vinay Shukla added a comment - Alejandro Abdelnur What is the audit story about the Key? Do we record, who did various key operations? I couldn't find it in the attached PDFs.
          Hide
          Alejandro Abdelnur added a comment -

          Owen,

          You need to allow the user to specify the key-name when they create the key.

          It is happening, it goes in the JSON payload of the create operation. Look in the createKeyInternal() method

          You never answered whether the body is xml or json (or both).

          Request/response bodies are JSON.

          I'll integrate your other 2 suggestions, the underscore and the break into multiple requests if URL goes beyond 2K byte limit.

          Show
          Alejandro Abdelnur added a comment - Owen, You need to allow the user to specify the key-name when they create the key. It is happening, it goes in the JSON payload of the create operation. Look in the createKeyInternal() method You never answered whether the body is xml or json (or both). Request/response bodies are JSON. I'll integrate your other 2 suggestions, the underscore and the break into multiple requests if URL goes beyond 2K byte limit.
          Hide
          Owen O'Malley added a comment -

          Alejandro,
          A couple more things on the interface:

          • You need to allow the user to specify the key-name when they create the key.
          • Instead of throwing if the url is too long, please break the user's request into parts that fit within the 2000 byte limit.
          • You never answered whether the body is xml or json (or both).
          • The underscores should be on the trailing part of the url.

          I'd propose it look like:

          create key         : PUT    http://HOST:PORT/kms/v1/key/<key-name>
          rollover key       : POST   http://HOST:PORT/kms/v1/key/<key-name>
          delete key         : DELETE http://HOST:PORT/kms/v1/key/<key-name>
          get metadata       : GET    http://HOST:PORT/kms/v1/key/<key-name>/_metadata
          get current version: GET    http://HOST:PORT/kms/v1/key/<key-name>/_currentversion
          get key versions   : GET    http://HOST:PORT/kms/v1/key/<keyname-name>/_versions
          get key version    : GET    http://HOST:PORT/kms/v1/keyversion/<version-name>
          get key names      : GET    http://HOST:PORT/kms/v1/keys/names
          get keys metadata  : GET    http://HOST:PORT/kms/v1/keys/metadata?key=<key-name>&key=<keyname>,...
          
          Show
          Owen O'Malley added a comment - Alejandro, A couple more things on the interface: You need to allow the user to specify the key-name when they create the key. Instead of throwing if the url is too long, please break the user's request into parts that fit within the 2000 byte limit. You never answered whether the body is xml or json (or both). The underscores should be on the trailing part of the url. I'd propose it look like: create key : PUT http: //HOST:PORT/kms/v1/key/<key-name> rollover key : POST http: //HOST:PORT/kms/v1/key/<key-name> delete key : DELETE http: //HOST:PORT/kms/v1/key/<key-name> get metadata : GET http: //HOST:PORT/kms/v1/key/<key-name>/_metadata get current version: GET http: //HOST:PORT/kms/v1/key/<key-name>/_currentversion get key versions : GET http: //HOST:PORT/kms/v1/key/<keyname-name>/_versions get key version : GET http: //HOST:PORT/kms/v1/keyversion/<version-name> get key names : GET http: //HOST:PORT/kms/v1/keys/names get keys metadata : GET http: //HOST:PORT/kms/v1/keys/metadata?key=<key-name>&key=<keyname>,...
          Hide
          Alejandro Abdelnur added a comment -

          sure, i'll wait till Friday. Thanks again.

          Show
          Alejandro Abdelnur added a comment - sure, i'll wait till Friday. Thanks again.
          Hide
          Andrew Wang added a comment -

          +1 from me, thanks Tucu. Really nice work here. Let's wait a bit before committing to see if anyone else still has review comments.

          Show
          Andrew Wang added a comment - +1 from me, thanks Tucu. Really nice work here. Let's wait a bit before committing to see if anyone else still has review comments.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642548/HADOOP-10433.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3878//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3878//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642548/HADOOP-10433.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3878//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3878//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          fixing typo in findbugs entry.

          Show
          Alejandro Abdelnur added a comment - fixing typo in findbugs entry.
          Hide
          Alejandro Abdelnur added a comment -

          Vinay, KMS uses a KeyProvider as the underlying keystore. A KeyProvider impl supporting RSA/SafeNet can be plugged in. AFAIK, both RSA and SafeNet have JCE providers with keystore support, thus getting the JavaKeyProvider to work with them shouldn't be that difficult (I would assume).

          Show
          Alejandro Abdelnur added a comment - Vinay, KMS uses a KeyProvider as the underlying keystore. A KeyProvider impl supporting RSA/SafeNet can be plugged in. AFAIK, both RSA and SafeNet have JCE providers with keystore support, thus getting the JavaKeyProvider to work with them shouldn't be that difficult (I would assume).
          Hide
          Larry McCay added a comment -

          KMS leverages KeyProvider API a kmip provider would be the integration vehicle. KMS would pick it up for free.

          Show
          Larry McCay added a comment - KMS leverages KeyProvider API a kmip provider would be the integration vehicle. KMS would pick it up for free.
          Hide
          Vinay Shukla added a comment -

          Does KMS support KMIP protocol ? (https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=kmip)

          Will KMS integrate with Hardware Security Modules (HSM) devices such as SafeNet Luna & RSA's Data Protection Manager (http://www.emc.com/security/rsa-data-protection-manager.htm)?

          If KMS does not speak KMIP, how can Hadoop encryption leverage enterprise grade Key management investment many enterprise level customers already have?

          Show
          Vinay Shukla added a comment - Does KMS support KMIP protocol ? ( https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=kmip ) Will KMS integrate with Hardware Security Modules (HSM) devices such as SafeNet Luna & RSA's Data Protection Manager ( http://www.emc.com/security/rsa-data-protection-manager.htm)? If KMS does not speak KMIP, how can Hadoop encryption leverage enterprise grade Key management investment many enterprise level customers already have?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642512/HADOOP-10433.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3874//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3874//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3874//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642512/HADOOP-10433.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3874//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3874//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3874//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment - - edited

          new patch changing PUT to POST for createKey and plus some naming refactoring on the client side classes.

          Show
          Alejandro Abdelnur added a comment - - edited new patch changing PUT to POST for createKey and plus some naming refactoring on the client side classes.
          Hide
          Alejandro Abdelnur added a comment -

          Larry, good point, i'll update the patch making it a POST. and I'll take care of the findbugs

          Show
          Alejandro Abdelnur added a comment - Larry, good point, i'll update the patch making it a POST. and I'll take care of the findbugs
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642504/HADOOP-10433.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3873//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3873//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3873//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642504/HADOOP-10433.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3873//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3873//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3873//console This message is automatically generated.
          Hide
          Larry McCay added a comment -

          Yes, that aligns with collections based actions - although they are supposed to be a POST for collection actions that create a new element. A PUT is supposed to replace the entire collection with another collection - which is NOT what you want.

          Element based actions on the other hand are done with a PUT and replace the element or create it if it does not exist.

          So, it seems that create should either be made a POST or an element based action.

          Show
          Larry McCay added a comment - Yes, that aligns with collections based actions - although they are supposed to be a POST for collection actions that create a new element. A PUT is supposed to replace the entire collection with another collection - which is NOT what you want. Element based actions on the other hand are done with a PUT and replace the element or create it if it does not exist. So, it seems that create should either be made a POST or an element based action.
          Hide
          Alejandro Abdelnur added a comment -

          Andrew, Owen, Larry, thanks for the reviews. New patch addressing them.

          Andrew,

          • KMS_SILENT does not suppress Tomcat startup messages now.
          • KMSMetadata & KMSKeyVersion classes are needed because their parent classes constructors are protected and per discussion in HADOOP-10431 there was not agreement on making them public.
          • Created a KMSClientRESTJSONUtils for client side REST constants/util methods
          • Created a KMSServerJSONUtils for server side JSON methods.
          • Renamed KMSClientProvider#getKeyMetadata to getMetadata, it is used on the client side on the corresponding client method.
          • on missing audits, we are not missing audits, all exceptions are caught, process and audit by the KMSAuthenticationFilter. It is a bit weird, but if we do it in the KMServer we still have to do it in the KMSAuthenticationFilter for uncaught exceptions and for jersey ones.
          • made all helper methods private static
          • On the validate not throwing exceptions other than IOException, fixed that, thrown the exception we got in the error if avail in the client, else an IOException
          • annotated KMSClientProvider#buildVersionName as @VisibleForTesting for now
          • Fixed the GET/GET_OP thing.
          • Added serialVersionUID = 1L to KeyNotFoundException (btw, this is Eclipse non-sense, this is not used at all outside of java serialization)
          • Using a local var cause in KMSCacheKeyProvider not to call ex.getCause() multiple times
          • Renamed KMSServer to KMS

          Owen,

          • I’ve added a check on the KMSClientProvider#getKeysMetadata() method to throw an IOException if the resulting URL is longer than 2K. I’d prefer to key the keys in the URL as using a HEADER would not be really REST.
          • I’m not sure where you are suggesting to use _ in fields, in the JSON fields?

          Larry,

          • Yes, the <key-name> is part o the JSON payload when creating the key. The create action is on the collection resource 'keys'. The Response "Location" header has the full URL to the created resource. AFAIK, this follows common REST practices.
          Show
          Alejandro Abdelnur added a comment - Andrew, Owen, Larry, thanks for the reviews. New patch addressing them. Andrew, KMS_SILENT does not suppress Tomcat startup messages now. KMSMetadata & KMSKeyVersion classes are needed because their parent classes constructors are protected and per discussion in HADOOP-10431 there was not agreement on making them public. Created a KMSClientRESTJSONUtils for client side REST constants/util methods Created a KMSServerJSONUtils for server side JSON methods. Renamed KMSClientProvider#getKeyMetadata to getMetadata, it is used on the client side on the corresponding client method. on missing audits, we are not missing audits, all exceptions are caught, process and audit by the KMSAuthenticationFilter. It is a bit weird, but if we do it in the KMServer we still have to do it in the KMSAuthenticationFilter for uncaught exceptions and for jersey ones. made all helper methods private static On the validate not throwing exceptions other than IOException, fixed that, thrown the exception we got in the error if avail in the client, else an IOException annotated KMSClientProvider#buildVersionName as @VisibleForTesting for now Fixed the GET/GET_OP thing. Added serialVersionUID = 1L to KeyNotFoundException (btw, this is Eclipse non-sense, this is not used at all outside of java serialization) Using a local var cause in KMSCacheKeyProvider not to call ex.getCause() multiple times Renamed KMSServer to KMS Owen, I’ve added a check on the KMSClientProvider#getKeysMetadata() method to throw an IOException if the resulting URL is longer than 2K. I’d prefer to key the keys in the URL as using a HEADER would not be really REST. I’m not sure where you are suggesting to use _ in fields, in the JSON fields? Larry, Yes, the <key-name> is part o the JSON payload when creating the key. The create action is on the collection resource 'keys'. The Response "Location" header has the full URL to the created resource. AFAIK, this follows common REST practices.
          Hide
          Larry McCay added a comment -

          I can see how having a .../keyversion/<version-name> is aligned with the KeyProvider API and that it allows the version-name to be more easily considered opaque.

          I would like to point out that I think we need to have the <key-name> for the create key API - you generally fully identify the resource that you are creating (PUTting). Are you assuming that it is part of the body of the request?

          Show
          Larry McCay added a comment - I can see how having a .../keyversion/<version-name> is aligned with the KeyProvider API and that it allows the version-name to be more easily considered opaque. I would like to point out that I think we need to have the <key-name> for the create key API - you generally fully identify the resource that you are creating (PUTting). Are you assuming that it is part of the body of the request?
          Hide
          Owen O'Malley added a comment -

          Alejandro,
          I like the direction your api is going. I'm a little concerned about the length of the URL (2000 bytes) causing trouble for the keys/metadata method. I'd suggest adding a field in the header that provides the list of key names. I'd also suggest that you add a character to the fields (metadata, currentversion, versions) such as adding underscore (_metadata, _currentversion, _versions).

          Show
          Owen O'Malley added a comment - Alejandro, I like the direction your api is going. I'm a little concerned about the length of the URL (2000 bytes) causing trouble for the keys/metadata method. I'd suggest adding a field in the header that provides the list of key names. I'd also suggest that you add a character to the fields (metadata, currentversion, versions) such as adding underscore (_metadata, _currentversion, _versions).
          Hide
          Larry McCay added a comment -

          Hey Alejandro Abdelnur - I was wondering about your " It may barbaric, but that is how things are and we should make sure we can work with those."

          I think you may have misinterpreted my reference the the OpenStack Barbican project - which is a REST based KMS server. I don't think UUIDs are barbaric.

          Thank's for the response.
          I need to give the rest of your explanation a bit more thought.

          Show
          Larry McCay added a comment - Hey Alejandro Abdelnur - I was wondering about your " It may barbaric, but that is how things are and we should make sure we can work with those." I think you may have misinterpreted my reference the the OpenStack Barbican project - which is a REST based KMS server. I don't think UUIDs are barbaric. Thank's for the response. I need to give the rest of your explanation a bit more thought.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642395/HADOOP-10433.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

          org.apache.hadoop.mapreduce.lib.db.TestDBJob
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

          The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

          org.apache.hadoop.mapred.pipes.TestPipeApplication

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3866//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3866//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642395/HADOOP-10433.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapreduce.lib.db.TestDBJob org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.pipes.TestPipeApplication +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3866//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3866//console This message is automatically generated.
          Hide
          Andrew Wang added a comment -

          Thanks for the rev here Tucu, I can tell this wasn't a small effort. Overall it looks real nice, the new REST API is fine by me. I have some more review comments though, but expect to +1 after this round (pending agreement by reviewers on the REST API details).

          Running the KMS:

          • Overall a more pleasant experience, tarball builds, audit logging has newlines, I didn't see stacktraces when running commands.
          • I know I asked for less spew in kms.sh, but defaulting KMS_SILENT to true is perhaps too little. Without it, running just "./sbin/kms.sh" shows nothing, not even help text, and if start/stop hit an error, we again see nothing (have to know where the logs are and hope it has the error there). This isn't a biggie, but maybe we can improve it a little more.

          JSON:

          • I see KMSMetadata and KMSKeyVersion classes, but they don't look like they do much. I'm guessing this is from an aborted attempt to do POJO<->JSON? That'd would be awesome, but I won't get too excited.
          • If you stick with the JSON helper methods, could we pull them all out into a separate helper class and make them all static, along with the field constants? Feels weird to have the Server digging into ClientProvider for field names, when theoretically a different client implementation (REST!) could be interfacing with the KMS.

          KMSServer and KMSClientProvider:

          • Noticed that KMSServer#getKeyMetadata includes the name, but it's unused on the client side, nor part of KeyProvider#Metadata.
          • I think we still miss audit logging if an exception is thrown. e.g. in #rolloverKey, getPrincipal, checkNotEmpty, provider.rollNewVersion can all throw something. This is why a try/finally might be easiest.
          • Can make many methods static: validateResponse, call, all the JSON functions, getPrincipal, assertAccess, removeKeyMaterial, getKeyURI
          • On the NoSuchAlgorithmException, #validateResponse casts everything to an IOException, so I don't think anything else comes out of this function. Not sure this cast is kosher. I think the catch for Exception there also needs to wrap the created String into an exception and throw it.
          • Given the discussion about UUIDs as versions, should we be exposing KMSClientProvider#buildVersionName publicly? This seems like an implementation detail of the backing KeyProvider.

          Misc things:

          • I think a rogue regex changed GET to GET_OP across the entire hadoop repo, which is why we're picking up all these test changes, and quite possibly why we have some weird test failures.
          • KMSACLs, the new comment about in #run probably should go in #loadACLs, or be removed.
          • KeyNotFoundException serialVersionUID, this is something Eclipse is warning about, so it'd be nice to just add it.
          • In the cache provider, #getCurrentKey and getKeyVersion, could set the cause once in a temp var to avoid having to call getCause() repeatedly.
          • KMSServer semantically should still be renamed to KMServer or just KMS
          Show
          Andrew Wang added a comment - Thanks for the rev here Tucu, I can tell this wasn't a small effort. Overall it looks real nice, the new REST API is fine by me. I have some more review comments though, but expect to +1 after this round (pending agreement by reviewers on the REST API details). Running the KMS: Overall a more pleasant experience, tarball builds, audit logging has newlines, I didn't see stacktraces when running commands. I know I asked for less spew in kms.sh, but defaulting KMS_SILENT to true is perhaps too little. Without it, running just "./sbin/kms.sh" shows nothing, not even help text, and if start/stop hit an error, we again see nothing (have to know where the logs are and hope it has the error there). This isn't a biggie, but maybe we can improve it a little more. JSON: I see KMSMetadata and KMSKeyVersion classes, but they don't look like they do much. I'm guessing this is from an aborted attempt to do POJO<->JSON? That'd would be awesome, but I won't get too excited. If you stick with the JSON helper methods, could we pull them all out into a separate helper class and make them all static, along with the field constants? Feels weird to have the Server digging into ClientProvider for field names, when theoretically a different client implementation (REST!) could be interfacing with the KMS. KMSServer and KMSClientProvider: Noticed that KMSServer#getKeyMetadata includes the name, but it's unused on the client side, nor part of KeyProvider#Metadata. I think we still miss audit logging if an exception is thrown. e.g. in #rolloverKey, getPrincipal, checkNotEmpty, provider.rollNewVersion can all throw something. This is why a try/finally might be easiest. Can make many methods static: validateResponse, call, all the JSON functions, getPrincipal, assertAccess, removeKeyMaterial, getKeyURI On the NoSuchAlgorithmException, #validateResponse casts everything to an IOException, so I don't think anything else comes out of this function. Not sure this cast is kosher. I think the catch for Exception there also needs to wrap the created String into an exception and throw it. Given the discussion about UUIDs as versions, should we be exposing KMSClientProvider#buildVersionName publicly? This seems like an implementation detail of the backing KeyProvider. Misc things: I think a rogue regex changed GET to GET_OP across the entire hadoop repo, which is why we're picking up all these test changes, and quite possibly why we have some weird test failures. KMSACLs, the new comment about in #run probably should go in #loadACLs, or be removed. KeyNotFoundException serialVersionUID, this is something Eclipse is warning about, so it'd be nice to just add it. In the cache provider, #getCurrentKey and getKeyVersion, could set the cause once in a temp var to avoid having to call getCause() repeatedly. KMSServer semantically should still be renamed to KMServer or just KMS
          Hide
          Alejandro Abdelnur added a comment -

          patch fixing the 'chiper' typo

          Larry, trying to answer your comments/concerns:

          The get key versions : GET http://HOST:PORT/kms/v1/key/<version-name>/versions was a typo in my comments and in the code, it should have been get key versions : GET http://HOST:PORT/kms/v1/key/<key-name>/versions

          The problem with using GET http://HOST:PORT/kms/v1/keys/<key-name>/<version-name> to get a version is that we are assuming the implementation first looks for the <key-name> and then for the <version-name>. The implementation could have direct look ups using the <version-name>. If that is the case, then the implementation would have then to do an additional lookup to verify that the specified <key-name> matches with the <version-name> given. In your example http://HOST:PORT/kms/v1/keys/1234/1234, if the version-name (the second 1234) is globally unique, then asking for http://HOST:PORT/kms/v1/keys/abcd/1234 should give an error, thus it would require an additional lookup in the implementation.

          Also, I think we should make sure that the version-name is opaque and that it’s construction (<key-name>@<version-count>) is not part of the public API. As you indicated there are system that use UUIDs as the version. It may barbaric, but that is how things are and we should make sure we can work with those. I don’t think we should change the JavaKeyStoreProvider to use UUIDs, it can continue using <key-name>@<version-count>. We just have to make sure that impls can use an opaque value as <version-name>. I’ll verify how things are and open a JIRA if necessary to follow up with this.

          Does it make sense?

          Show
          Alejandro Abdelnur added a comment - patch fixing the 'chiper' typo Larry, trying to answer your comments/concerns: The get key versions : GET http://HOST:PORT/kms/v1/key/ <version-name>/versions was a typo in my comments and in the code, it should have been get key versions : GET http://HOST:PORT/kms/v1/key/ <key-name>/versions The problem with using GET http://HOST:PORT/kms/v1/keys/ <key-name>/<version-name> to get a version is that we are assuming the implementation first looks for the <key-name> and then for the <version-name>. The implementation could have direct look ups using the <version-name>. If that is the case, then the implementation would have then to do an additional lookup to verify that the specified <key-name> matches with the <version-name> given. In your example http://HOST:PORT/kms/v1/keys/1234/1234 , if the version-name (the second 1234) is globally unique, then asking for http://HOST:PORT/kms/v1/keys/abcd/1234 should give an error, thus it would require an additional lookup in the implementation. Also, I think we should make sure that the version-name is opaque and that it’s construction (<key-name>@<version-count>) is not part of the public API. As you indicated there are system that use UUIDs as the version. It may barbaric, but that is how things are and we should make sure we can work with those. I don’t think we should change the JavaKeyStoreProvider to use UUIDs, it can continue using <key-name>@<version-count>. We just have to make sure that impls can use an opaque value as <version-name>. I’ll verify how things are and open a JIRA if necessary to follow up with this. Does it make sense?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642355/HADOOP-10433.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

          org.apache.hadoop.mapreduce.lib.db.TestDBJob
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup

          The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

          org.apache.hadoop.mapred.pipes.TestPipeApplication

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3865//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3865//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642355/HADOOP-10433.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapreduce.lib.db.TestDBJob org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.pipes.TestPipeApplication +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3865//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3865//console This message is automatically generated.
          Hide
          Larry McCay added a comment -

          AFAICT - we don't have a problem for a key named currentversion or metadata. Currentversion would have special meaning within the versions context and metadata wouldn't make sense as a version name anyway.

          I don't see why a key and version name couldn't be the same either - though I think versions are generally numeric. So, key 1234 could have 1234 versions in which case 1234@1234 would be reasonable and not precluded by the URI's proposed.

          get key version    : GET    http://HOST:PORT/kms/v1/keys/1234/1234
          
          Show
          Larry McCay added a comment - AFAICT - we don't have a problem for a key named currentversion or metadata. Currentversion would have special meaning within the versions context and metadata wouldn't make sense as a version name anyway. I don't see why a key and version name couldn't be the same either - though I think versions are generally numeric. So, key 1234 could have 1234 versions in which case 1234@1234 would be reasonable and not precluded by the URI's proposed. get key version : GET http: //HOST:PORT/kms/v1/keys/1234/1234
          Hide
          Larry McCay added a comment -

          Hi Andrew Wang - I also thought that those types of providers would need to be adapted by the KeyProvider implementation to always default the version - assuming that the provider doesn't actually support versions. So, in that case it would be be UUID@0. Current version would always be 0 or 1 - whatever makes sense.

          There are other systems that seem to have keyNames with UUIDs as the version. I think that barbican uses this scheme. It would have to do something similar within the KeyProvider implementation to adapt it to the API. Not sure whether the UUID version can be used as the actual version.

          Show
          Larry McCay added a comment - Hi Andrew Wang - I also thought that those types of providers would need to be adapted by the KeyProvider implementation to always default the version - assuming that the provider doesn't actually support versions. So, in that case it would be be UUID@0. Current version would always be 0 or 1 - whatever makes sense. There are other systems that seem to have keyNames with UUIDs as the version. I think that barbican uses this scheme. It would have to do something similar within the KeyProvider implementation to adapt it to the API. Not sure whether the UUID version can be used as the actual version.
          Hide
          Larry McCay added a comment -

          hmmm...

          Seems like this one won't work that way with the rest:

          get key versions   : GET    http://HOST:PORT/kms/v1/keys/<version-name>/versions
          

          I'm not even really sure I know what it is trying to depict.
          Perhaps it is supposed to be like?:

          get key versions   : GET    http://HOST:PORT/kms/v1/keys/<key-name>/versions
          

          I still don't quite like the last one with .../keys/metadata as it overlaps with a key named metadata in the hierarchy.
          I'm not sure what to do with that one.

          Maybe the following would work - obviously you aren't asking for the names since you are providing them:

          get keys metadata  : GET    http://HOST:PORT/kms/v1/keys?key=<key-name>&key=<keyname>,...
          
          Show
          Larry McCay added a comment - hmmm... Seems like this one won't work that way with the rest: get key versions : GET http: //HOST:PORT/kms/v1/keys/<version-name>/versions I'm not even really sure I know what it is trying to depict. Perhaps it is supposed to be like?: get key versions : GET http: //HOST:PORT/kms/v1/keys/<key-name>/versions I still don't quite like the last one with .../keys/metadata as it overlaps with a key named metadata in the hierarchy. I'm not sure what to do with that one. Maybe the following would work - obviously you aren't asking for the names since you are providing them: get keys metadata : GET http: //HOST:PORT/kms/v1/keys?key=<key-name>&key=<keyname>,...
          Hide
          Andrew Wang added a comment -

          Larry, I think an issue with that scheme is that we can't have a key with the name "currentversion" or "metadata" then, or a key and a version with the same name. We need additional path elements to disambiguate.

          As to the "get key version" API, it's think it's an unfortunate consequence of the existing KeyProvider#getKeyVersion API, which takes a single string. Tucu's concern there is that if a KP decides to use a UUID instead of "keyName@version" like JavaKeyStoreProvider, we can't fit it into the hierarchical key-then-version scheme.

          Show
          Andrew Wang added a comment - Larry, I think an issue with that scheme is that we can't have a key with the name "currentversion" or "metadata" then, or a key and a version with the same name. We need additional path elements to disambiguate. As to the "get key version" API, it's think it's an unfortunate consequence of the existing KeyProvider#getKeyVersion API, which takes a single string. Tucu's concern there is that if a KP decides to use a UUID instead of "keyName@version" like JavaKeyStoreProvider, we can't fit it into the hierarchical key-then-version scheme.
          Hide
          Charles Lamb added a comment -

          Nit: there are still some instances of "chiper" in the design doc.

          Show
          Charles Lamb added a comment - Nit: there are still some instances of "chiper" in the design doc.
          Hide
          Larry McCay added a comment -

          I like this much better as well.
          To me, you could continue the hierarchy of the key/keyversion as well.
          Additionally, I'm not sure that you need to identify the path elements with key/keyname/version/version-name.
          Each resource's path is unique unto itself and I think it holds together to just use the values that way - as you follow the path the URI more specifically identifies the resource.

          With that in mind, I would propose adjustments along the following lines:

          create key         : PUT    http://HOST:PORT/kms/v1/keys
          rollover key       : POST   http://HOST:PORT/kms/v1/keys/<key-name>
          delete key         : DELETE http://HOST:PORT/kms/v1/keys/<key-name>
          get metadata       : GET    http://HOST:PORT/kms/v1/keys/<key-name>/metadata
          get current version: GET    http://HOST:PORT/kms/v1/keys/<key-name>/currentversion
          get key version    : GET    http://HOST:PORT/kms/v1/keys/<key-name>/<version-name>
          get key versions   : GET    http://HOST:PORT/kms/v1/keys/<version-name>/versions
          get key names      : GET    http://HOST:PORT/kms/v1/keys
          get keys metadata  : GET    http://HOST:PORT/kms/v1/keys/metadata?key=<key-name>&key=<keyname>,...
          

          This flattens the hierarchy a good bit the only outlier being the get .../keys/metadata - I think that the rest of them identify unique resources within a hierarchy.

          Alejandro Abdelnur
          I'm not sure that I am grasping your concern with the 'get key version' URI and doubt that this minor adjustments changes your feelings there. Can you elaborate on what the issue is? Maybe provide an example?

          Show
          Larry McCay added a comment - I like this much better as well. To me, you could continue the hierarchy of the key/keyversion as well. Additionally, I'm not sure that you need to identify the path elements with key/keyname/version/version-name. Each resource's path is unique unto itself and I think it holds together to just use the values that way - as you follow the path the URI more specifically identifies the resource. With that in mind, I would propose adjustments along the following lines: create key : PUT http: //HOST:PORT/kms/v1/keys rollover key : POST http: //HOST:PORT/kms/v1/keys/<key-name> delete key : DELETE http: //HOST:PORT/kms/v1/keys/<key-name> get metadata : GET http: //HOST:PORT/kms/v1/keys/<key-name>/metadata get current version: GET http: //HOST:PORT/kms/v1/keys/<key-name>/currentversion get key version : GET http: //HOST:PORT/kms/v1/keys/<key-name>/<version-name> get key versions : GET http: //HOST:PORT/kms/v1/keys/<version-name>/versions get key names : GET http: //HOST:PORT/kms/v1/keys get keys metadata : GET http: //HOST:PORT/kms/v1/keys/metadata?key=<key-name>&key=<keyname>,... This flattens the hierarchy a good bit the only outlier being the get .../keys/metadata - I think that the rest of them identify unique resources within a hierarchy. Alejandro Abdelnur I'm not sure that I am grasping your concern with the 'get key version' URI and doubt that this minor adjustments changes your feelings there. Can you elaborate on what the issue is? Maybe provide an example?
          Hide
          Charles Lamb added a comment -

          I like this better.

          Show
          Charles Lamb added a comment - I like this better.
          Hide
          Alejandro Abdelnur added a comment -

          New patch with the following REST API:

          create key         : PUT    http://HOST:PORT/kms/v1/keys
          rollover key       : POST   http://HOST:PORT/kms/v1/key/<key-name>
          delete key         : DELETE http://HOST:PORT/kms/v1/key/<key-name>
          get metadata       : GET    http://HOST:PORT/kms/v1/key/<key-name>/metadata
          get current version: GET    http://HOST:PORT/kms/v1/key/<key-name>/currentversion
          get key version    : GET    http://HOST:PORT/kms/v1/keyversion/<version-name>
          get key versions   : GET    http://HOST:PORT/kms/v1/key/<version-name>/versions
          get key names      : GET    http://HOST:PORT/kms/v1/keys/names
          get keys metadata  : GET    http://HOST:PORT/kms/v1/keys/metadata?key=<key-name>&key=<keyname>,...
          

          not entirely happy with the 'get key version' URL but using http://HOST:PORT/kms/v1/key/<keyname>/version/<version-name> seems wrong as it would preclude implementations of KeyProvider that use opaque key versions (like an UUID).

          The test-patch failing testcases seem unrelated.

          Show
          Alejandro Abdelnur added a comment - New patch with the following REST API: create key : PUT http: //HOST:PORT/kms/v1/keys rollover key : POST http: //HOST:PORT/kms/v1/key/<key-name> delete key : DELETE http: //HOST:PORT/kms/v1/key/<key-name> get metadata : GET http: //HOST:PORT/kms/v1/key/<key-name>/metadata get current version: GET http: //HOST:PORT/kms/v1/key/<key-name>/currentversion get key version : GET http: //HOST:PORT/kms/v1/keyversion/<version-name> get key versions : GET http: //HOST:PORT/kms/v1/key/<version-name>/versions get key names : GET http: //HOST:PORT/kms/v1/keys/names get keys metadata : GET http: //HOST:PORT/kms/v1/keys/metadata?key=<key-name>&key=<keyname>,... not entirely happy with the 'get key version' URL but using http://HOST:PORT/kms/v1/key/ <keyname>/version/<version-name> seems wrong as it would preclude implementations of KeyProvider that use opaque key versions (like an UUID). The test-patch failing testcases seem unrelated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642256/HADOOP-10433.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

          org.apache.hadoop.mapreduce.lib.db.TestDBJob
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

          The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

          org.apache.hadoop.mapred.pipes.TestPipeApplication

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3863//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3863//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642256/HADOOP-10433.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapreduce.lib.db.TestDBJob org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.pipes.TestPipeApplication +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3863//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3863//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          ok, let me work on a revised version using path to refer sub-resources metadata/version/versions

          Show
          Alejandro Abdelnur added a comment - ok, let me work on a revised version using path to refer sub-resources metadata/version/versions
          Hide
          Larry McCay added a comment -

          I do have to say that I don't like the use of "get" as a query parameter. It is semantically redundant with the HTTP verb in a well designed REST API.

          Show
          Larry McCay added a comment - I do have to say that I don't like the use of "get" as a query parameter. It is semantically redundant with the HTTP verb in a well designed REST API.
          Hide
          Larry McCay added a comment -

          Personally, I feel that the keyname as part of the URI adheres to accepted REST principles. In fact, I'd like to understand why the parameters are needed at all. One should be able to represent the desired resource entirely by URI. There are times when non-hierarchical attributes need to be considered (as filters) that are easily added to a query string. But largely, the APIs we are looking at here are hierarchical.

          GET .../keys - would return all the keyNames
          GET .../keys/keyname - would return the metadata for that key (or the current key maybe)
          GET .../keys/keyname/version - would return the specific key (could have a well-known "current" version as well)

          Just my two cents.

          Show
          Larry McCay added a comment - Personally, I feel that the keyname as part of the URI adheres to accepted REST principles. In fact, I'd like to understand why the parameters are needed at all. One should be able to represent the desired resource entirely by URI. There are times when non-hierarchical attributes need to be considered (as filters) that are easily added to a query string. But largely, the APIs we are looking at here are hierarchical. GET .../keys - would return all the keyNames GET .../keys/keyname - would return the metadata for that key (or the current key maybe) GET .../keys/keyname/version - would return the specific key (could have a well-known "current" version as well) Just my two cents.
          Hide
          Alejandro Abdelnur added a comment -

          well, typically REST resources are expressed as a PATH more than as QUERY_STRING, I'd prefer to keep it that way.

          Show
          Alejandro Abdelnur added a comment - well, typically REST resources are expressed as a PATH more than as QUERY_STRING, I'd prefer to keep it that way.
          Hide
          Charles Lamb added a comment -

          I guess what I was responding to was having keynames embedded in the URI rather than as a parameter after the ?. For instance, the get key version operation uses

          /<key-name>?get=version

          but the get keys metadata operation puts the keyname as a parameter value:

          /keys?get=keyMetaData&key=<key-name>

          I like the second form where the key-name is a value rather than an operator so perhaps this can be unified?

          Show
          Charles Lamb added a comment - I guess what I was responding to was having keynames embedded in the URI rather than as a parameter after the ?. For instance, the get key version operation uses /<key-name>?get=version but the get keys metadata operation puts the keyname as a parameter value: /keys?get=keyMetaData&key=<key-name> I like the second form where the key-name is a value rather than an operator so perhaps this can be unified?
          Hide
          Alejandro Abdelnur added a comment -

          Charles, based on Andrew's feedback I've moved away from operations. get indicates what part of the resource we want to retrieve, either get= or show= I would say, op= is a bit confusing as it is not an operation.

          Show
          Alejandro Abdelnur added a comment - Charles, based on Andrew's feedback I've moved away from operations. get indicates what part of the resource we want to retrieve, either get= or show= I would say, op= is a bit confusing as it is not an operation.
          Hide
          Charles Lamb added a comment -

          One little nit in the REST API: for the get operations, rather than use "?get=...", perhaps consider calling it "op" instead. For example,

          GET .../key/<key-name>?op=getCurrentVersion

          and

          GET .../keys?op=getKeyNames

          Show
          Charles Lamb added a comment - One little nit in the REST API: for the get operations, rather than use "?get=...", perhaps consider calling it "op" instead. For example, GET .../key/<key-name>?op=getCurrentVersion and GET .../keys?op=getKeyNames
          Hide
          Alejandro Abdelnur added a comment -

          Tweak to TestKMSServer testACLs() to fix reload flakiness.

          Show
          Alejandro Abdelnur added a comment - Tweak to TestKMSServer testACLs() to fix reload flakiness.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642105/HADOOP-10433.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

          org.apache.hadoop.ha.TestZKFailoverControllerStress
          org.apache.hadoop.mapreduce.lib.db.TestDBJob
          org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

          The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

          org.apache.hadoop.mapred.pipes.TestPipeApplication

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3858//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3858//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642105/HADOOP-10433.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.ha.TestZKFailoverControllerStress org.apache.hadoop.mapreduce.lib.db.TestDBJob org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.pipes.TestPipeApplication +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3858//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3858//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642105/HADOOP-10433.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

          org.apache.hadoop.crypto.key.kms.server.TestKMSServer
          org.apache.hadoop.mapreduce.lib.db.TestDBJob
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

          The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

          org.apache.hadoop.mapred.pipes.TestPipeApplication

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3859//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3859//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642105/HADOOP-10433.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.crypto.key.kms.server.TestKMSServer org.apache.hadoop.mapreduce.lib.db.TestDBJob org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.pipes.TestPipeApplication +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3859//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3859//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642080/HADOOP-10433.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

          org.apache.hadoop.mapreduce.lib.db.TestDBJob
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

          The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

          org.apache.hadoop.mapred.pipes.TestPipeApplication

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3857//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3857//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-kms.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3857//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642080/HADOOP-10433.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapreduce.lib.db.TestDBJob org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.pipes.TestPipeApplication +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3857//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3857//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-kms.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3857//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          fixing findbugs, other failures seem unrelated

          Show
          Alejandro Abdelnur added a comment - fixing findbugs, other failures seem unrelated
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642061/HADOOP-10433.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

          org.apache.hadoop.mapreduce.lib.db.TestDBJob
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

          The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

          org.apache.hadoop.mapred.pipes.TestPipeApplication

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3856//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3856//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-kms.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3856//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642061/HADOOP-10433.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapreduce.lib.db.TestDBJob org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.pipes.TestPipeApplication +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3856//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3856//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-kms.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3856//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          reattaching patch as test-patch picked up the PDF

          Show
          Alejandro Abdelnur added a comment - reattaching patch as test-patch picked up the PDF
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642042/HadoopKMSDocsv2.pdf
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3853//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642042/HadoopKMSDocsv2.pdf against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3853//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          New patch addressing Andrew’s feedback.

          The documentation defines the HTTP REST API (attaching PDF of it for easier review)

          The ACLs have been changed to be able to discriminate per KeyProvider operation.

          BTW, the audits logs in the KMSServer are done at the end of every method, once we know the method is successful, for failures, the KMSAuthenticationFilter will audit all failures.

          On the comment not using Hadoop ACLs, I would prefer using it and then improve it (in a diff JIRA) to suppport ":" as separator.

          Show
          Alejandro Abdelnur added a comment - New patch addressing Andrew’s feedback. The documentation defines the HTTP REST API (attaching PDF of it for easier review) The ACLs have been changed to be able to discriminate per KeyProvider operation. BTW, the audits logs in the KMSServer are done at the end of every method, once we know the method is successful, for failures, the KMSAuthenticationFilter will audit all failures. On the comment not using Hadoop ACLs, I would prefer using it and then improve it (in a diff JIRA) to suppport ":" as separator.
          Hide
          Owen O'Malley added a comment -

          Please don't use the Hadoop ACLs. They are very error-prone with "x" meaning user "x" and " x" meaning group x. I'd suggest using ":" as the separator and making it ignore spaces.

          Show
          Owen O'Malley added a comment - Please don't use the Hadoop ACLs. They are very error-prone with "x" meaning user "x" and " x" meaning group x. I'd suggest using ":" as the separator and making it ignore spaces.
          Hide
          Owen O'Malley added a comment -

          What is the form of the urls for the REST service? What are the acceptable formats of the data in the REST service (JSON? XML?)?

          Show
          Owen O'Malley added a comment - What is the form of the urls for the REST service? What are the acceptable formats of the data in the REST service (JSON? XML?)?
          Hide
          Alejandro Abdelnur added a comment -

          Andrew, thanks for the detailed review.

          I'm working on a patch addressing most of your feedback, following some questions/clarifications:

          • KeyNotFoundException needing serialVersionUID, why do we need this? we are not serializing these exceptions.
          • The big log entries happen only on the first REST call only, a Jersey initialization thing (fixed in newer Jerseys)
          • Regarding using curl directly, the REST API is not meant to be public at the moment, that is why it is not documented.
          • Doing POJO<->JSON automatically requires the beans to have a public default constructor and public getters/setters. Given the discussion in HADOOP-10431, this is not an option.
          • KMSClientProvider constants going private, all of them are used in the KMSServer as well.
          • KMSClientProvider.rollNewVersion() throwing a NoSuchAlgorithmException, yes it can, it may be come as part of the remote exception in the error message.
          Show
          Alejandro Abdelnur added a comment - Andrew, thanks for the detailed review. I'm working on a patch addressing most of your feedback, following some questions/clarifications: KeyNotFoundException needing serialVersionUID, why do we need this? we are not serializing these exceptions. The big log entries happen only on the first REST call only, a Jersey initialization thing (fixed in newer Jerseys) Regarding using curl directly, the REST API is not meant to be public at the moment, that is why it is not documented. Doing POJO<->JSON automatically requires the beans to have a public default constructor and public getters/setters. Given the discussion in HADOOP-10431 , this is not an option. KMSClientProvider constants going private, all of them are used in the KMSServer as well. KMSClientProvider.rollNewVersion() throwing a NoSuchAlgorithmException, yes it can, it may be come as part of the remote exception in the error message.
          Hide
          Andrew Wang added a comment -

          Thanks for the huge amount of work here Tucu. I'm still fresh to this topic, so please forgive any silly review comments.

          • In the docs, "Hadoop KMS Server" is like saying "ATM machine" docs could use a pass, but we can do that after the base patch is committed.
          • For the same reason, you could rename "KMSServer" to just "KMS" or "KMServer"
          • More javadoc, at the minimum class javadoc for everything
          • KeyNotFoundException needs a serialVersionUID
          • pom.xml, not sure where these vars are being used, but I'm not aware of Maven's subtleties.
                <kms.source.repository>REPO NOT AVAIL</kms.source.repository>
                <kms.source.revision>REVISION NOT AVAIL</kms.source.revision>
                <kms.build.timestamp>${maven.build.timestamp}</kms.build.timestamp>
            
          • "mvn package -Pdist -Dtar" doesn't work because there's an extra newline in the pom.xml for dist-maketar.sh.

          Notes from running the KMS:

          • Running "./sbin/kms.sh" has a lot of debugging output, not sure if we can quash it or not.
          • Is there some way of aborting tomcat if the KMSWebApp dies? I didn't put a core-site.xml in my conf dir initially, KMSWebApp FATAL'd, and I had a useless Tomcat process lying around.
          • Running ops against the KMS with "hadoop key" generates big log entries like this:
            2014-04-16 16:40:28,160 INFO  WadlGeneratorJAXBGrammarGenerator - Couldn't find JAX-B element for class javax.ws.rs.core.Response
            2014-04-16 16:40:28,160 ERROR WadlGeneratorJAXBGrammarGenerator - 
            java.lang.IllegalAccessException: Class com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 can not access a member of class javax.ws.rs.core.Response with modifiers "protected"
            <stacktrace>
            
          • Audit logging doesn't have newlines
          • I didn't test anything outside of KeyShell operations. I couldn't figure out the right curl incantation to hit REST endpoints directly, whenever I tried I got an authentication exception about being an anonymous user.

          REST API design comments:

          • It's kind of ugly that we're doing manual serde of things into JSON, since it throws away types and makes the protocol harder to understand. Jackson has ways of automatically doing POJO<->JSON for beans, which seems pretty reasonable to me.
          • If we use Jackson, we should really aspire to JSON in, JSON out instead of using so many query parameters (in particular, create).
          • I think we should bake the op into the path when possible, since it makes the API more self-descriptive. Maybe something like this:
            /keys
            /keys?metadata # this is still ugly, dunno
            /keys/<keyName> # GET for metadata, PUT for create, POST for roll, DELETE
            /keys/<keyName>/versions # getKeyVersions
            /keys/<keyName>/versions/<version> # getKeyVersion
            /keys/<keyName>/versions/current
            
          • Also, shouldn't the REST API be explained in the docs? I assume we chose REST for interop with non-Hadoop and non-Java components, so the protocol should be documented.

          KMSClientProvider:

          • Class javadoc refers to JksProviders
          • Typo: CHIPER_FIELD and chiper, I assume cipher
          • It'd be nice to put all the private static final before public static final. Some of the public ones also aren't used outside of this class, so we could close them up.
          • Constructor, can use equalsIgnoreCase rather than toLowerCase
          • Wherever we're building a URL, consider using URIBuilder and then toURL() instead.
          • Based on this page (http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html) we should try/catch when calling HttpURLConnection#getInputStream and I assume #getResponseCode too, and also close the input stream after we're done. The error stream logic in #validateResponse also doesn't quite adhere to the recommendations.
          • Is there a reason flush is a NOP? The JKSP doesn't do this, so a explanatory comment would help.
          • It doesn't look like rollNewVersionInternal needs to throw NoSuchAlgorithmException?

          KMSACLs:

          • You could write this with a ScheduledExecutorService and a ThreadFactoryBuilder, more idiomatic.
          • Probably should make start/stopReloader idempotent to future proof it, e.g. check a "isRunning" boolean.
          • In loadACLs, I assume you do the conf get to init the conf outside of the lock? Would help to clarify that in the comment.
          • The lastReload time should be set to the modtime of the file, not by calling System.currentTimeMillis again. Also, what's the point of waiting for it to be at least 100ms newer?
          • I'd bump the setACLs logs up to INFO, since this probably isn't a frequent operation and is pretty useful.

          KMSServer:

          • The hasAccess method is a bit gross. We could make an Access enum rather than passing the ACL as an argument. Could also get rid of the boolean for throwing an exception, and have two helper methods checkAccess and assertAccess or something like that.
          • I think it'd also be clearer if all the audit logging happened in each function via a try/finally. The control flow with Exceptions means that right now we don't always log something.
          • Another q, why is the audit logger trying to log client method names rather the API that the server provides, e.g. logging "getKeyVersion()" rather than KMSClientProvider.GET_VERSION? Having these strings sprinkled everywhere also seems prone to typos.
          • It'd also be better to have the throws state the specific exceptions, rather than just throwing Exception.
          • For createKey, why is everything being passed as a query parameter rather than a JSON blob in the payload?
          • If we get rid of OP by putting it in the path, we could get rid of a bunch of these IllegalArgumentExceptions since it'll just 404 instead. Could also use Preconditions.checkArgument instead to be more concise.

          Tests:

          • Got these errors when running the tests, can attach the full logs if you like:
            ailed tests: 
              TestKMSServer.testACLs:505->runServer:142->access$0:257->doAs:270 java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 401, message: Authentication required
              TestKMSServer.testServicePrincipalACLs:958->runServer:142->access$0:257->doAs:270 java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 401, message: Authentication required
            
            Tests in error: 
              TestKMSServer.testStartStopHttpKerberos:345->testStartStop:304->runServer:142->access$0:257->doAs:270 » PrivilegedAction
              TestKMSServer.testStartStopHttpsKerberos:350->testStartStop:304->runServer:142->access$0:257->doAs:270 » PrivilegedAction
            
          Show
          Andrew Wang added a comment - Thanks for the huge amount of work here Tucu. I'm still fresh to this topic, so please forgive any silly review comments. In the docs, "Hadoop KMS Server" is like saying "ATM machine" docs could use a pass, but we can do that after the base patch is committed. For the same reason, you could rename "KMSServer" to just "KMS" or "KMServer" More javadoc, at the minimum class javadoc for everything KeyNotFoundException needs a serialVersionUID pom.xml, not sure where these vars are being used, but I'm not aware of Maven's subtleties. <kms.source.repository>REPO NOT AVAIL</kms.source.repository> <kms.source.revision>REVISION NOT AVAIL</kms.source.revision> <kms.build.timestamp>${maven.build.timestamp}</kms.build.timestamp> "mvn package -Pdist -Dtar" doesn't work because there's an extra newline in the pom.xml for dist-maketar.sh. Notes from running the KMS: Running "./sbin/kms.sh" has a lot of debugging output, not sure if we can quash it or not. Is there some way of aborting tomcat if the KMSWebApp dies? I didn't put a core-site.xml in my conf dir initially, KMSWebApp FATAL'd, and I had a useless Tomcat process lying around. Running ops against the KMS with "hadoop key" generates big log entries like this: 2014-04-16 16:40:28,160 INFO WadlGeneratorJAXBGrammarGenerator - Couldn't find JAX-B element for class javax.ws.rs.core.Response 2014-04-16 16:40:28,160 ERROR WadlGeneratorJAXBGrammarGenerator - java.lang.IllegalAccessException: Class com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 can not access a member of class javax.ws.rs.core.Response with modifiers " protected " <stacktrace> Audit logging doesn't have newlines I didn't test anything outside of KeyShell operations. I couldn't figure out the right curl incantation to hit REST endpoints directly, whenever I tried I got an authentication exception about being an anonymous user. REST API design comments: It's kind of ugly that we're doing manual serde of things into JSON, since it throws away types and makes the protocol harder to understand. Jackson has ways of automatically doing POJO<->JSON for beans, which seems pretty reasonable to me. If we use Jackson, we should really aspire to JSON in, JSON out instead of using so many query parameters (in particular, create). I think we should bake the op into the path when possible, since it makes the API more self-descriptive. Maybe something like this: /keys /keys?metadata # this is still ugly, dunno /keys/<keyName> # GET for metadata, PUT for create, POST for roll, DELETE /keys/<keyName>/versions # getKeyVersions /keys/<keyName>/versions/<version> # getKeyVersion /keys/<keyName>/versions/current Also, shouldn't the REST API be explained in the docs? I assume we chose REST for interop with non-Hadoop and non-Java components, so the protocol should be documented. KMSClientProvider: Class javadoc refers to JksProviders Typo: CHIPER_FIELD and chiper, I assume cipher It'd be nice to put all the private static final before public static final . Some of the public ones also aren't used outside of this class, so we could close them up. Constructor, can use equalsIgnoreCase rather than toLowerCase Wherever we're building a URL, consider using URIBuilder and then toURL() instead. Based on this page ( http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html ) we should try/catch when calling HttpURLConnection#getInputStream and I assume #getResponseCode too, and also close the input stream after we're done. The error stream logic in #validateResponse also doesn't quite adhere to the recommendations. Is there a reason flush is a NOP? The JKSP doesn't do this, so a explanatory comment would help. It doesn't look like rollNewVersionInternal needs to throw NoSuchAlgorithmException? KMSACLs: You could write this with a ScheduledExecutorService and a ThreadFactoryBuilder, more idiomatic. Probably should make start/stopReloader idempotent to future proof it, e.g. check a "isRunning" boolean. In loadACLs, I assume you do the conf get to init the conf outside of the lock? Would help to clarify that in the comment. The lastReload time should be set to the modtime of the file, not by calling System.currentTimeMillis again. Also, what's the point of waiting for it to be at least 100ms newer? I'd bump the setACLs logs up to INFO, since this probably isn't a frequent operation and is pretty useful. KMSServer: The hasAccess method is a bit gross. We could make an Access enum rather than passing the ACL as an argument. Could also get rid of the boolean for throwing an exception, and have two helper methods checkAccess and assertAccess or something like that. I think it'd also be clearer if all the audit logging happened in each function via a try/finally. The control flow with Exceptions means that right now we don't always log something. Another q, why is the audit logger trying to log client method names rather the API that the server provides, e.g. logging "getKeyVersion()" rather than KMSClientProvider.GET_VERSION ? Having these strings sprinkled everywhere also seems prone to typos. It'd also be better to have the throws state the specific exceptions, rather than just throwing Exception . For createKey, why is everything being passed as a query parameter rather than a JSON blob in the payload? If we get rid of OP by putting it in the path, we could get rid of a bunch of these IllegalArgumentExceptions since it'll just 404 instead. Could also use Preconditions.checkArgument instead to be more concise. Tests: Got these errors when running the tests, can attach the full logs if you like: ailed tests: TestKMSServer.testACLs:505->runServer:142->access$0:257->doAs:270 java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 401, message: Authentication required TestKMSServer.testServicePrincipalACLs:958->runServer:142->access$0:257->doAs:270 java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 401, message: Authentication required Tests in error: TestKMSServer.testStartStopHttpKerberos:345->testStartStop:304->runServer:142->access$0:257->doAs:270 » PrivilegedAction TestKMSServer.testStartStopHttpsKerberos:350->testStartStop:304->runServer:142->access$0:257->doAs:270 » PrivilegedAction
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12639861/HADOOP-10433.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3788//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3788//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12639861/HADOOP-10433.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3788//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3788//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          patch fixing javac warning.

          Show
          Alejandro Abdelnur added a comment - patch fixing javac warning.
          Hide
          Larry McCay added a comment -

          Alejandro Abdelnur - that sounds like the right direction to me!

          Show
          Larry McCay added a comment - Alejandro Abdelnur - that sounds like the right direction to me!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12639842/HADOOP-10433.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          -1 javac. The applied patch generated 1289 javac compiler warnings (more than the trunk's current 1288 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3787//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3787//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3787//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12639842/HADOOP-10433.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. -1 javac . The applied patch generated 1289 javac compiler warnings (more than the trunk's current 1288 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3787//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3787//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3787//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          Larry McCay the KMS server is meant to be a proxy to any KeyProvider implementation with the following benefits:

          • Single entry point from Hadoop to the KeyProvider implementation
          • Supports Hadoop Kerberos Authentication, making secure integration easier
          • Provides caching, to support load from Hadoop services and jobs
          Show
          Alejandro Abdelnur added a comment - Larry McCay the KMS server is meant to be a proxy to any KeyProvider implementation with the following benefits: Single entry point from Hadoop to the KeyProvider implementation Supports Hadoop Kerberos Authentication, making secure integration easier Provides caching, to support load from Hadoop services and jobs
          Hide
          Alejandro Abdelnur added a comment -

          updated patch.

          Show
          Alejandro Abdelnur added a comment - updated patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12639468/COMBO.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified test files.

          -1 javac. The applied patch generated 1289 javac compiler warnings (more than the trunk's current 1288 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist:

          org.apache.hadoop.crypto.key.TestKeyShell

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3775//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3775//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3775//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12639468/COMBO.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 8 new or modified test files. -1 javac . The applied patch generated 1289 javac compiler warnings (more than the trunk's current 1288 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist: org.apache.hadoop.crypto.key.TestKeyShell +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3775//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3775//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3775//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          combo patch including HADOOP-10428/HADOOP-10430/HADOOP-10431/HADOOP-10433 for test-patch to patch successfully.

          Show
          Alejandro Abdelnur added a comment - combo patch including HADOOP-10428 / HADOOP-10430 / HADOOP-10431 / HADOOP-10433 for test-patch to patch successfully.
          Hide
          Alejandro Abdelnur added a comment -

          updated patch based on changes in HADOOP-10430

          Show
          Alejandro Abdelnur added a comment - updated patch based on changes in HADOOP-10430
          Hide
          Larry McCay added a comment -

          I don't think that features are an issue at this point.
          My larger concern is whether Hadoop common needs a KMS server offering of its own or not.
          Given that the keystore provider is not likely a scalable provider for a KMS that will lead us down the path of requiring an appropriate DB to do it right. This was my thinking as I starting prototyping on an earlier version of the KeyProvider API as well.

          It became reasonable to me that the KeyProvider API within Hadoop common was sufficient to enable deployments to plugin any number of various KeyProvider implementations. This might be:
          1. direct to KMIP
          2. to an OpenStack Barbican server
          3. to a Knox (or wherever) KMS

          This would allow Hadoop common to not have to take on the weight of another server.

          I'm not really arguing against putting it common here but recollecting my thought process on the matter.

          Here is a question that I have had trouble answering with regard to the server and API being in common together...

          Given the pluggable nature of the KeyProvider API what value does adding a full KMS server with additional (and maybe redundant) pluggability to common provide?

          I've had trouble reconciling that in my mind.

          Now, take that same server implementation and move it out of common and it easily makes sense to be able to have pluggability for the Hadoop KeyProvider API, KMIP and others.

          So, when I said "as long as it make sense to do so", I was really talking about whether it made sense to have it colocated with the KeyProvider API in common. I think the feature set is less of an issue.

          Show
          Larry McCay added a comment - I don't think that features are an issue at this point. My larger concern is whether Hadoop common needs a KMS server offering of its own or not. Given that the keystore provider is not likely a scalable provider for a KMS that will lead us down the path of requiring an appropriate DB to do it right. This was my thinking as I starting prototyping on an earlier version of the KeyProvider API as well. It became reasonable to me that the KeyProvider API within Hadoop common was sufficient to enable deployments to plugin any number of various KeyProvider implementations. This might be: 1. direct to KMIP 2. to an OpenStack Barbican server 3. to a Knox (or wherever) KMS This would allow Hadoop common to not have to take on the weight of another server. I'm not really arguing against putting it common here but recollecting my thought process on the matter. Here is a question that I have had trouble answering with regard to the server and API being in common together... Given the pluggable nature of the KeyProvider API what value does adding a full KMS server with additional (and maybe redundant) pluggability to common provide? I've had trouble reconciling that in my mind. Now, take that same server implementation and move it out of common and it easily makes sense to be able to have pluggability for the Hadoop KeyProvider API, KMIP and others. So, when I said "as long as it make sense to do so", I was really talking about whether it made sense to have it colocated with the KeyProvider API in common. I think the feature set is less of an issue.
          Hide
          Alejandro Abdelnur added a comment -

          Larry McCay, if you have additional features in mind we should incorporate them in the KMS, agree. Would make sense to do them as follow up improvements/new-features JIRAs?

          Regarding HA, the KMS server itself is stateless, you can put multiple behind a VIP and are done (from the KMS server perspective). And the KeyProvider impl you are using must be HA itself (for example, it could be a HA DB or a HA KMIP sever).

          Uma Maheswara Rao G, regarding Tomcat vs Jetty. With embedded Jetty we will have to create configurations for all sorts of HTTP and server related things (connection threadpool size, http/socket timeout, ssl, port, etc, etc). With Tomcat all that come for free. The same approach is taken by HttpFS (and Oozie).

          Show
          Alejandro Abdelnur added a comment - Larry McCay , if you have additional features in mind we should incorporate them in the KMS, agree. Would make sense to do them as follow up improvements/new-features JIRAs? Regarding HA, the KMS server itself is stateless, you can put multiple behind a VIP and are done (from the KMS server perspective). And the KeyProvider impl you are using must be HA itself (for example, it could be a HA DB or a HA KMIP sever). Uma Maheswara Rao G , regarding Tomcat vs Jetty. With embedded Jetty we will have to create configurations for all sorts of HTTP and server related things (connection threadpool size, http/socket timeout, ssl, port, etc, etc). With Tomcat all that come for free. The same approach is taken by HttpFS (and Oozie).
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Alejandro, Thanks a lot, for the doc.
          Seems like you plan to use Tomcat which will be bundled with Hadoop. Is there any special reason for not using Jetty which we are already using for simple web in Hadoop components?
          Yes, As pointed by Larry, I would like to see how the HA requirements would be handled.

          Show
          Uma Maheswara Rao G added a comment - Hi Alejandro, Thanks a lot, for the doc. Seems like you plan to use Tomcat which will be bundled with Hadoop. Is there any special reason for not using Jetty which we are already using for simple web in Hadoop components? Yes, As pointed by Larry, I would like to see how the HA requirements would be handled.
          Hide
          Larry McCay added a comment -

          I have a lot of interest in this project.
          We have done some prototyping for a REST based key server within Knox.
          I'd like to hopefully consolidate these into this jira as long as it makes sense to do so.

          Obviously, there are HA requirements that aren't described here yet that will need some thought and discussion.

          Show
          Larry McCay added a comment - I have a lot of interest in this project. We have done some prototyping for a REST based key server within Knox. I'd like to hopefully consolidate these into this jira as long as it makes sense to do so. Obviously, there are HA requirements that aren't described here yet that will need some thought and discussion.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12637950/KMS-ALL-PATCHES-v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3731//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3731//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637950/KMS-ALL-PATCHES-v3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3731//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3731//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          3rd is the charm. KMS combo patch.

          Show
          Alejandro Abdelnur added a comment - 3rd is the charm. KMS combo patch.
          Hide
          Alejandro Abdelnur added a comment -

          3rd is the charm. HADOOP-10433 patch.

          Show
          Alejandro Abdelnur added a comment - 3rd is the charm. HADOOP-10433 patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12637934/KMS-ALL-PATCHES-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          -1 javac. The applied patch generated 1487 javac compiler warnings (more than the trunk's current 1485 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3730//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3730//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3730//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637934/KMS-ALL-PATCHES-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. -1 javac . The applied patch generated 1487 javac compiler warnings (more than the trunk's current 1485 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3730//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3730//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3730//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          new combo for test-patch.

          Show
          Alejandro Abdelnur added a comment - new combo for test-patch.
          Hide
          Alejandro Abdelnur added a comment -

          addressing test-patch complains.

          Show
          Alejandro Abdelnur added a comment - addressing test-patch complains.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12637338/KMS-ALL-PATCHES.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          -1 javac. The applied patch generated 1504 javac compiler warnings (more than the trunk's current 1491 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist:

          org.apache.hadoop.crypto.key.kms.server.TestKMSACLs
          org.apache.hadoop.crypto.key.kms.server.TestKMSServer

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3726//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3726//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-kms.html
          Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3726//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3726//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637338/KMS-ALL-PATCHES.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. -1 javac . The applied patch generated 1504 javac compiler warnings (more than the trunk's current 1491 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist: org.apache.hadoop.crypto.key.kms.server.TestKMSACLs org.apache.hadoop.crypto.key.kms.server.TestKMSServer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3726//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3726//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-kms.html Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3726//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3726//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          Attaching a combo patch (including all required JIRAs) so people can play with it (and to run test-patch as all the required JIRAs are still not committed)

          Show
          Alejandro Abdelnur added a comment - Attaching a combo patch (including all required JIRAs) so people can play with it (and to run test-patch as all the required JIRAs are still not committed)
          Hide
          Alejandro Abdelnur added a comment -

          Patch with a KMS server implementation with the following features:

          • A HTTP REST API on top of the KeyProvider API
          • KMS Server proxies to any KeyProvider implementation
          • for HTTP/S transport
          • Kerberos HTTP SPNEGO authentication
          • 3 different ACLs to guard operations (admin, get-key-material, accept-user-key-material)
          • Audit log for all operations, succesful/failed-attempts
          • Key caching to handle burst of request without impacting underlying KeyProvider
          • scripts to start/stop the KMS server

          It uses a Tomcat server to run the KMS server. Includes documentation (PDF of it attached)

          Show
          Alejandro Abdelnur added a comment - Patch with a KMS server implementation with the following features: A HTTP REST API on top of the KeyProvider API KMS Server proxies to any KeyProvider implementation for HTTP/S transport Kerberos HTTP SPNEGO authentication 3 different ACLs to guard operations (admin, get-key-material, accept-user-key-material) Audit log for all operations, succesful/failed-attempts Key caching to handle burst of request without impacting underlying KeyProvider scripts to start/stop the KMS server It uses a Tomcat server to run the KMS server. Includes documentation (PDF of it attached)

            People

            • Assignee:
              Alejandro Abdelnur
              Reporter:
              Alejandro Abdelnur
            • Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development