[PARQUET-1414] Limit page size based on maximum row count

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.11.0
• Labels: pull-request-available
• Assignee: Gabor Szadovszky
• Reporter: Gabor Szadovszky
• Created: 07/Sep/18 11:56
• Updated: 23/Jun/24 03:30
• Resolved: 07/Nov/18 08:45

Description

For column index based filtering it is important to have enough pages for a column. In case of a perfectly matching encoding for the suitable data it can happen that all of the values can be encoded in one page (e.g. a column of an ascending counter).

With this improvement we would be able to limit the pages by the maximum number of rows to be written in it so we would have enough pages for every column.

Based on the benchmarks listed here 20k seems to be a good choice for the default value.

Issue Links

• is related to: PARQUET-1201 Column indexes
• is related to: IMPALA-8449 Avoid Parquet pages with too many rows + try to make them aligned
• is related to: IMPALA-10405 Consider setting parquet_page_row_count_limit to 20000 by default
• links to: GitHub Pull Request #531
• links to: GitHub Pull Request #537
• links to: Page size limit to file size test results