[IMPALA-7936] Enable better control over Parquet writing

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 3.3.0
• Epic Color: ghx-label-6
• Assignee: Zoltán Borók-Nagy
• Reporter: Zoltán Borók-Nagy
• Created: 06/Dec/18 18:28
• Updated: 22/Dec/20 22:29
• Resolved: 22/May/19 09:17

Description

With the introduction of the Parquet page indexes it became desirable to have more control over how Impala writes Parquet files.

These configuration options (probably implemented as query options) would be:

• enable/disable Parquet page index writing (currently we can do it with a command-line argument)
• set page-size limits based on row count
• Set truncation length for statistics about string values   (current truncation length is 64, it is unlikely to have user data that needs longer truncation than that)

They'd enable writing more complete tests for page filtering. They'd be also useful for fine-tuning the page index for better performance.

Issue Links

• is related to: IMPALA-8449 Avoid Parquet pages with too many rows + try to make them aligned
• is related to: IMPALA-10405 Consider setting parquet_page_row_count_limit to 20000 by default

Sub-Tasks

• Impala Doc: New query options for Parquet writing: Closed