[PARQUET-922] Add index pages to the format to support efficient page skipping

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: format-2.4.0
• Component/s: parquet-format
• Assignee: Marcel Kinard
• Reporter: Julien Le Dem
• Created: 24/Mar/17 18:45
• Updated: 23/Jun/24 03:29
• Resolved: 16/Oct/17 23:49

Description

When a Parquet file is sorted we can define an index consisting of the boundary values for the pages of the columns sorted on as well as the offsets and length of said pages in the file.
The goal is to optimize lookup and range scan type queries, using this to read only the pages containing data matching the filter.
We'd require the pages to be aligned accross columns.

marcelk will add a link to the google doc to discuss the spec

Issue Links

• blocks: PARQUET-1134 Release Parquet format 2.4.0
• is depended upon by: PARQUET-1201 Column indexes
• is depended upon by: PARQUET-1207 Write index page in parquet file
• is required by: IMPALA-5840 Don't write page level statistics in Parquet files in anticipation of page indexes
• is required by: IMPALA-5842 Write page index in Parquet files
• links to: Format PR #72
• links to: GitHub Pull Request #63
• links to: Impala prototype implementation results