[ORC-14] Add column level encryption to ORC files

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Assignee: Owen O'Malley
• Reporter: Owen O'Malley
• Created: 25/Mar/13 17:14
• Updated: 04/Sep/19 02:00
• Resolved: 30/Aug/19 22:11

Description

It would be useful to support column level encryption in ORC files. Since each column and its associated index is stored separately, encrypting a column separately isn't difficult. In terms of key distribution, it would make sense to use an external server like the one in HADOOP-9331.

Issue Links

• is a parent of: HIVE-21848 Table property name definition between ORC and Parquet encrytion
• is related to: HADOOP-9331 Hadoop crypto codec framework and crypto codec implementations
• relates to: HADOOP-9534 Credential Management Framework (CMF)

Sub-Tasks

• Create framework for data masking.: Closed
• Create sha256 mask: Closed
• Modify InStream and OutStream to optionally encrypt data: Closed
• Add support for Key Management Servers (kms) to HadoopShims: Closed
• Add unmasked ranges option for redact mask: Closed
• Create in memory KeyProvider class: Closed
• Write documentation for column encryption: Open
• Change HadoopShims.KeyProvider to separate createLocalKey and decryptLocalKey: Closed
• Separate the compression options from the CompressionCodec: Closed
• Extend specification and protobuf definition with column encryption: Closed
• Extend physicalwriter for encryption: Closed
• Add the API changes for getting column encryption: Closed
• Refactor the TreeWriter and WriterContext APIs so that TreeWriters can deal with encryption: Closed
• Extend the stripe read planner to understand encryption: Closed
• Update the KeyProvider to match spec: Closed
• Cleanup API for StreamOptions and CompressionCodec.Options: Closed
• Modify InStream for column encryption: Closed
• Fix file merging for column encryption.: Closed
• Update metadata tools to print encryption information.: Open
• Update ReaderImpl to support column encryption: Closed
• Add support for table properties to control column encryption: Closed