[HADOOP-10150] Hadoop cryptographic file system

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 2.6.0
• Component/s: security
• Labels: rhino
• Target Version/s: fs-encryption (HADOOP-10150 and HDFS-6134)
• Assignee: Yi Liu
• Reporter: Yi Liu
• Created: 29/Aug/13 07:54
• Updated: 12/May/16 18:24
• Resolved: 20/Aug/14 18:58

Description

There is an increasing need for securing data when Hadoop customers use various upper layer applications, such as Map-Reduce, Hive, Pig, HBase and so on.

HADOOP CFS (HADOOP Cryptographic File System) is used to secure data, based on HADOOP “FilterFileSystem” decorating DFS or other file systems, and transparent to upper layer applications. It’s configurable, scalable and fast.

High level requirements:
1. Transparent to and no modification required for upper layer applications.
2. “Seek”, “PositionedReadable” are supported for input stream of CFS if the wrapped file system supports them.
3. Very high performance for encryption and decryption, they will not become bottleneck.
4. Can decorate HDFS and all other file systems in Hadoop, and will not modify existing structure of file system, such as namenode and datanode structure if the wrapped file system is HDFS.
5. Admin can configure encryption policies, such as which directory will be encrypted.
6. A robust key management framework.
7. Support Pread and append operations if the wrapped file system supports them.

Issue Links

• breaks: HADOOP-11286 Map/Reduce dangerously adds Guava @Beta class to CryptoUtils
• is related to: HADOOP-9534 Credential Management Framework (CMF)
• is related to: HDFS-6134 Transparent data at rest encryption
• requires: HDFS-2006 ability to support storing extended attributes per file

Sub-Tasks

• Crypto input and output streams implementing Hadoop stream interfaces: Resolved
• Tests for Crypto input and output streams using fake streams implementing Hadoop streams interfaces.: Resolved
• Javadoc and few code style improvement for Crypto input and output streams: Resolved
• Minor improvements to Crypto input and output streams: Closed
• Add a method to CryptoCodec to generate SRNs for IV: Closed
• Add a new constructor for CryptoInputStream that receives current position of wrapped stream.: Resolved
• NullPointerException in CryptoInputStream while wrapped stream is not ByteBufferReadable. Add tests using normal stream.: Resolved
• Implementation of AES-CTR CryptoCodec using JNI to OpenSSL: Resolved
• Refactor CryptoCodec#generateSecureRandom to take a byte[]: Resolved
• Implement high-performance secure random number sources: Resolved
• Fall back AesCtrCryptoCodec implementation from OpenSSL to JCE if non native support.: Resolved
• UnsatisfiedLinkError in cryptocodec tests with OpensslCipher#initContext: Resolved
• Update OpensslCipher#getInstance to accept CipherSuite#name format.: Resolved
• Refactor get instance of CryptoCodec and support create via algorithm/mode/padding.: Resolved
• Failed to load OpenSSL cipher error logs on systems with old openssl versions: Resolved
• incorrect prototype in OpensslSecureRandom.c: Resolved
• CryptoCodec#getCodecclasses throws NPE when configurations not loaded.: Closed