Hadoop Common
  1. Hadoop Common
  2. HADOOP-10433

Key Management Server based on KeyProvider API

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 2.6.0
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      (from HDFS-6134 proposal)

      Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality).

      Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177.

      Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation.

      The client-server protocol will be secure:

      • Kerberos HTTP SPNEGO (authentication)
      • HTTPS for transport (confidentiality and integrity)
      • Hadoop ACLs (authorization)

      The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used.

      Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool

      There are minor changes that must be done in Hadoop KeyProvider functionality:

      The KeyProvider contract, and the existing implementations, must be thread-safe

      KeyProvider API should have an API to generate the key material internally
      JavaKeyStoreProvider should use, if present, a password provided via configuration

      KeyProvider Option and Metadata should include a label (for easier cross-referencing)

      To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy.

      Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available.

      1. HADOOP-10433.patch
        192 kB
        Alejandro Abdelnur
      2. HADOOP-10433.patch
        191 kB
        Alejandro Abdelnur
      3. HADOOP-10433.patch
        191 kB
        Alejandro Abdelnur
      4. HADOOP-10433.patch
        191 kB
        Alejandro Abdelnur
      5. HADOOP-10433.patch
        203 kB
        Alejandro Abdelnur
      6. HADOOP-10433.patch
        203 kB
        Alejandro Abdelnur
      7. HADOOP-10433.patch
        202 kB
        Alejandro Abdelnur
      8. HADOOP-10433.patch
        201 kB
        Alejandro Abdelnur
      9. HADOOP-10433.patch
        201 kB
        Alejandro Abdelnur
      10. HADOOP-10433.patch
        201 kB
        Alejandro Abdelnur
      11. HadoopKMSDocsv2.pdf
        527 kB
        Alejandro Abdelnur
      12. HADOOP-10433.patch
        183 kB
        Alejandro Abdelnur
      13. HADOOP-10433.patch
        183 kB
        Alejandro Abdelnur
      14. KMS-doc.pdf
        237 kB
        Alejandro Abdelnur

        Issue Links

          Activity

          Alejandro Abdelnur created issue -
          Alejandro Abdelnur made changes -
          Field Original Value New Value
          Link This issue is blocked by HADOOP-10432 [ HADOOP-10432 ]
          Alejandro Abdelnur made changes -
          Link This issue is blocked by HADOOP-10429 [ HADOOP-10429 ]
          Alejandro Abdelnur made changes -
          Link This issue is blocked by HADOOP-10428 [ HADOOP-10428 ]
          Alejandro Abdelnur made changes -
          Link This issue is blocked by HADOOP-10427 [ HADOOP-10427 ]
          Alejandro Abdelnur made changes -
          Link This issue is blocked by HADOOP-10430 [ HADOOP-10430 ]
          Alejandro Abdelnur made changes -
          Link This issue is blocked by HADOOP-10431 [ HADOOP-10431 ]
          Alejandro Abdelnur made changes -
          Summary Key Management Server base on KeyProvider API Key Management Server based on KeyProvider API
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12637335 ]
          Attachment KMS-doc.pdf [ 12637336 ]
          Alejandro Abdelnur made changes -
          Attachment KMS-ALL-PATCHES.patch [ 12637338 ]
          Alejandro Abdelnur made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433-v2.patch [ 12637933 ]
          Alejandro Abdelnur made changes -
          Attachment KMS-ALL-PATCHES-v2.patch [ 12637934 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433-v3.patch [ 12637949 ]
          Alejandro Abdelnur made changes -
          Attachment KMS-ALL-PATCHES-v3.patch [ 12637950 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12639467 ]
          Alejandro Abdelnur made changes -
          Attachment COMBO.patch [ 12639468 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12639842 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12639861 ]
          Alejandro Abdelnur made changes -
          Attachment COMBO.patch [ 12639468 ]
          Alejandro Abdelnur made changes -
          Attachment KMS-ALL-PATCHES.patch [ 12637338 ]
          Alejandro Abdelnur made changes -
          Attachment KMS-ALL-PATCHES-v2.patch [ 12637934 ]
          Alejandro Abdelnur made changes -
          Attachment KMS-ALL-PATCHES-v3.patch [ 12637950 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433-v3.patch [ 12637949 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433-v2.patch [ 12637933 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12637335 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12639467 ]
          Xiaomeng Huang made changes -
          Issue Type Improvement [ 4 ] New Feature [ 2 ]
          Xiaomeng Huang made changes -
          Issue Type New Feature [ 2 ] Improvement [ 4 ]
          Larry McCay made changes -
          Link This issue is duplicated by HADOOP-10528 [ HADOOP-10528 ]
          Owen O'Malley made changes -
          Link This issue is blocked by HADOOP-10534 [ HADOOP-10534 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12642041 ]
          Attachment HadoopKMSDocsv2.pdf [ 12642042 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12642061 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12642041 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12642080 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12642105 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12642256 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12642355 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12642395 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12642504 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12642512 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12642548 ]
          Alejandro Abdelnur made changes -
          Attachment HADOOP-10433.patch [ 12642989 ]
          Alejandro Abdelnur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 3.0.0 [ 12320357 ]
          Resolution Fixed [ 1 ]
          Alejandro Abdelnur made changes -
          Fix Version/s 2.6.0 [ 12327179 ]
          Fix Version/s 3.0.0 [ 12320357 ]
          Chris Nauroth made changes -
          Link This issue breaks HADOOP-11063 [ HADOOP-11063 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Alejandro Abdelnur
              Reporter:
              Alejandro Abdelnur
            • Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development