[HDFS-6134] Transparent data at rest encryption

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0, 3.0.0-alpha1
• Fix Version/s: 2.6.0
• Component/s: security
• Target Version/s: fs-encryption (HADOOP-10150 and HDFS-6134)
• Assignee: Charles Lamb
• Reporter: Alejandro Abdelnur
• Created: 21/Mar/14 06:08
• Updated: 14/Nov/22 11:58
• Resolved: 20/Aug/14 19:07

Description

Because of privacy and security regulations, for many industries, sensitive data at rest must be in encrypted form. For example: the health­care industry (HIPAA regulations), the card payment industry (PCI DSS regulations) or the US government (FISMA regulations).

This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can be used transparently by any application accessing HDFS via Hadoop Filesystem Java API, Hadoop libhdfs C library, or WebHDFS REST API.

The resulting implementation should be able to be used in compliance with different regulation requirements.

Issue Links

• breaks: HADOOP-11286 Map/Reduce dangerously adds Guava @Beta class to CryptoUtils
• is related to: HADOOP-10919 Copy command should preserve raw.* namespace extended attributes
• is related to: MAPREDUCE-6007 Add support to distcp to preserve raw.* namespace extended attributes
• relates to: HADOOP-9333 Hadoop crypto codec framework based on compression codec
• relates to: HDFS-6891 Follow-on work for transparent data at rest encryption
• relates to: HADOOP-10141 Create an API to separate encryption key storage from applications
• relates to: HADOOP-10150 Hadoop cryptographic file system
• requires: HADOOP-10856 HarFileSystem and HarFs support for HDFS encryption

Sub-Tasks

• HDFS Encryption Zones: Resolved
• HDFS integration with KeyProvider: Resolved
• Wire crypto streams for encrypted files in DFSClient: Resolved
• Protocol and API for Encryption Zones: Resolved
• Print out the KeyProvider after finding KP successfully on startup: Resolved
• CryptoCode.generateSecureRandom should be a static method: Resolved
• HDFS CLI admin tool for creating & deleting an encryption zone: Resolved
• Get the Key/IV from the NameNode for encrypted files in DFSClient: Resolved
• Rename restrictions for encryption zones: Resolved
• Client server negotiation of cipher suite: Resolved
• Remove the Delete Encryption Zone function: Resolved
• List of Encryption Zones should be based on inodes: Resolved
• Test Crypto streams in HDFS: Resolved
• Namenode needs to get the actual keys and iv from the KeyProvider: Resolved
• Clean up encryption-related tests: Resolved
• Fix the keyid format for generated keys in FSNamesystem.createEncryptionZone: Resolved
• Not able to create symlinks after HDFS-6516: Resolved
• Refactor encryption zone functionality into new EncryptionZoneManager class: Resolved
• Update usage of KeyProviderCryptoExtension APIs on NameNode: Resolved
• Remove EncryptionZoneManager lock: Resolved
• Remove unnecessary getEncryptionZoneForPath call in EZManager#createEncryptionZone: Resolved
• Remove KeyProvider in EncryptionZoneManager: Resolved
• Decrypt EDEK before creating CryptoInputStream/CryptoOutputStream: Resolved
• Creating encryption zone results in NPE when KeyProvider is null: Resolved
• Create a special /.reserved/raw directory for raw access to encrypted data: Resolved
• Create a .RAW extended attribute namespace: Resolved
• Add more HDFS encryption tests: Resolved
• Should not be able to create encryption zone using path to a non-directory file: Resolved
• Require specification of an encryption key when creating an encryption zone: Resolved
• Batch the encryption zones listing API: Resolved
• DFSClient should use IV generated based on the configured CipherSuite with codecs used: Resolved
• Cannot remove directory within encryption zone to Trash: Resolved
• Fix TestReservedRawPaths failures: Resolved
• Mistakenly dfs.namenode.list.encryption.zones.num.responses configured as boolean: Resolved
• HDFS encryption documentation: Resolved
• Fix findbugs and other warnings: Resolved
• Improve the configuration guidance in DFSClient when there are no Codec classes found in configs: Resolved
• Fix TestCLI to expect new output: Resolved
• Add non-superuser capability to get the encryption zone for a specific path: Resolved
• Constants in CommandWithDestination should be static: Resolved