[PARQUET-1872] Add TransCompression Feature

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.0
• Component/s: parquet-mr
• Assignee: Xinli Shang
• Reporter: Xinli Shang
• Created: 12/Jun/20 03:41
• Updated: 14/Jan/22 21:32
• Resolved: 14/Jan/22 21:32

Description

When ZSTD becomes more popular, there is a need to translate existing data to ZSTD compressed which can achieve a higher compression ratio. It would be useful if we can have a tool to convert a Parquet file directly by just decompressing/compressing each page without decoding/encoding or assembling the record because it is much faster. The initial result shows it is ~5 times faster.

Issue Links

• Blocked: PARQUET-1949 Mark Parquet-1872 with not support bloom filter yet
• links to: PR-796

Sub-Tasks

• Add to Parquet-tools: Resolved
• Add to parquet-cli: Resolved
• Add bloom filter support: Open