[ARROW-13593] [C++][Dataset][Parquet] Support parquet modular encryption in the new Dataset API

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: C++, Parquet
• External issue URL: https://github.com/apache/arrow/issues/29238
• Assignee: Unassigned
• Reporter: Maya Anderson
• Created: 10/Aug/21 08:00
• Updated: 11/Jan/23 08:34

Description

In order for the new Dataset API to fully support PME, the same writer properties that include file_encryption_properties shouldn’t be used for the whole dataset. file_encryption_properties should be per file, for example in order to support key rotation ARROW-9960 .
Design document: https://docs.google.com/document/d/13EysCNC6-Nu9wnJ8YpdzmD-aMLn4i2KXUJTNqIihy7A/edit#

 

Issue Links

• Dependency: ARROW-9960 [C++][Parquet] Enable external material and rotation for encryption keys
• is depended upon by: ARROW-15986 [Python] Expose decryption properties in ParquetFragmentScanOptions
• supercedes: ARROW-17285 [Python] decryption_properties missing in pyarrow.parquet.ParquetDataset
• links to: Design document