[MAPREDUCE-4491] Encryption and Key Protection

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Won't Do
• Component/s: documentation, security, task-controller, tasktracker
• Assignee: Benoy Antony
• Reporter: Benoy Antony
• Created: 27/Jul/12 01:24
• Updated: 16/Oct/17 19:44
• Resolved: 16/Oct/17 19:44

Description

When dealing with sensitive data, it is required to keep the data encrypted wherever it is stored. Common use case is to pull encrypted data out of a datasource and store in HDFS for analysis. The keys are stored in an external keystore.

The feature adds a customizable framework to integrate different types of keystores, support for Java KeyStore, read keys from keystores, and transport keys from JobClient to Tasks.
The feature adds PGP encryption as a codec and additional utilities to perform encryption related steps.

The design document is attached. It explains the requirement, design and use cases.
Kindly review and comment. Collaboration is very much welcome.

I have a tested patch for this for 1.1 and will upload it soon as an initial work for further refinement.

Update: The patches are uploaded to subtasks.

Issue Links

• depends upon: MAPREDUCE-4554 Job Credentials are not transmitted if security is turned off

Sub-Tasks

• Key Protection : Define Encryption and Key Protection interfaces and default implementations: Resolved
• Key Protection : Add ability to read keys and protect keys in JobClient and TTS/NodeManagers: Resolved
• Encryption: Add support for PGP Encryption: Resolved
• Key Protection : Implement KeyProvider to read key from a WebService Based KeyStore: Resolved
• Key Distribution and Management for supporting crypto codec in Map Reduce: Open