[PARQUET-2075] Unified Rewriter Tool

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.13.0
• External issue URL: https://docs.google.com/document/d/1Ryt5uXnp-YwOrsnIDrGdTMoFfbOaTM6X39pBLHOa_50
• External issue ID: https://issues.apache.org/jira/browse/PARQUET-2071
• Assignee: Gang Wu
• Reporter: Xinli Shang
• Created: 05/Aug/21 15:00
• Updated: 23/Jun/24 03:32
• Resolved: 12/Apr/23 01:29

Description

During the discussion of PARQUET-2071, we came up with the idea of a universal tool to translate the existing file to a different state while skipping some level steps like encoding/decoding, to gain speed. For example, only decompress pages and then compress directly. For PARQUET-2071, we only decrypt and then encrypt directly. This will be useful for the existing data to onboard Parquet features like column encryption, zstd etc.

We already have tools like trans-compression, column pruning etc. We will consolidate all these tools with this universal tool.

Sub-Tasks

• Refactor different file rewriters to use single implementation: Resolved
• ParquetRewriter supports more than one input file: Resolved
• ParquetRewriter supports masking and encrypting the same column: Resolved
• Add a new rewrite command powered by ParquetRewriter: Resolved
• ParquetRewriter supports merging files by record: Open