Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.12.0
-
None
Description
Some personal data columns need to be masked instead of being pruned(Parquet-1791). We need a tool to replace the raw data columns with masked value. The masked value could be hash, null, redact etc. For the unchanged columns, they should be moved as a whole like 'merge', 'prune' command in Parquet-tools.
Implementing this feature in file format is 10X faster than doing it by rewriting the table data in the query engine.
Attachments
1.
|
Add null command | Resolved | Xinli Shang | |
2.
|
Add hash functionality | Open | Unassigned |