[HUDI-2370] Supports data encryption (COW tables)

• Type: New Feature
• Status: Closed
• Priority: Blocker
• Resolution: Done
• Fix Version/s: 0.11.0
• Labels: pull-request-available
• Story Points: 2
• Epic Link: Table data encryption
• Assignee: liujinhui
• Reporter: liujinhui

Reviewers

Raymond Xu, Vinoth Chandar

• Created: 27/Aug/21 08:52
• Updated: 21/Aug/22 05:36
• Resolved: 15/Feb/22 04:33

Description

Data security is becoming more and more important, if hudi can support encryption, it is very welcome

1. Specify column encryption
2. Support footer encryption
3. Custom encrypted client interface（Provide memory-based encryption client by default）
4. Specify the encryption key

 

When querying, you need to pass the relevant key or obtain query permission based on the client's encrypted interface. If it fails, the result cannot be returned.
1. When querying non-encrypted fields, the key is not passed, and the data is returned normally
2. When querying encrypted fields, the key is not passed and the data is not returned
3. When the encrypted field is queried, the key is passed, and the data is returned normally
4. When querying all fields, the key is not passed and no result is returned. If passed, the data returns normally

 

Start with COW first

Issue Links

• Blocked: HUDI-2594 Upgrade parquet version 1.12.1
• links to: GitHub Pull Request #3612
• links to: GitHub Pull Request #3614
• links to: GitHub Pull Request #4483
• links to: GitHub Pull Request #4804