[HADOOP-10433] Key Management Server based on KeyProvider API

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 2.6.0
• Component/s: security
• Hadoop Flags: Reviewed
• Assignee: Alejandro Abdelnur
• Reporter: Alejandro Abdelnur
• Created: 25/Mar/14 06:15
• Updated: 12/May/16 18:26
• Resolved: 05/May/14 21:43

Description

(from HDFS-6134 proposal)

Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality).

Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177.

Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation.

The client-server protocol will be secure:

• Kerberos HTTP SPNEGO (authentication)
• HTTPS for transport (confidentiality and integrity)
• Hadoop ACLs (authorization)

The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used.

Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool

There are minor changes that must be done in Hadoop KeyProvider functionality:

The KeyProvider contract, and the existing implementations, must be thread-safe

KeyProvider API should have an API to generate the key material internally
JavaKeyStoreProvider should use, if present, a password provided via configuration

KeyProvider Option and Metadata should include a label (for easier cross-referencing)

To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy.

Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available.

Issue Links

• breaks: HADOOP-11063 KMS cannot deploy on Windows, because class names are too long.
• is blocked by: HADOOP-10534 KeyProvider API should using windowing for retrieving metadata
• is blocked by: HADOOP-10427 KeyProvider implementations should be thread safe
• is blocked by: HADOOP-10428 JavaKeyStoreProvider should accept keystore password via configuration falling back to ENV VAR
• is blocked by: HADOOP-10429 KeyStores should have methods to generate the materials themselves, KeyShell should use them
• is blocked by: HADOOP-10430 KeyProvider Metadata should have an optional description, there should be a method to retrieve the metadata from all keys
• is blocked by: HADOOP-10431 Change visibility of KeyStore.Options getter methods to public
• is blocked by: HADOOP-10432 Refactor SSLFactory to expose static method to determine HostnameVerifier
• is duplicated by: HADOOP-10528 A TokenKeyProvider for a Centralized Key Manager Server (BEE: bee-key-manager)