[PARQUET-374] Add api to read dictionary from each column chunk for predicate pushdown

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: parquet-mr
• Assignee: Zhenxiao Luo
• Reporter: Zhenxiao Luo
• Created: 12/Sep/15 01:33
• Updated: 09/Mar/16 18:27
• Resolved: 09/Mar/16 18:27

Description

Parquet files's dictionary could be used for predicate pushdown
eg.
SQL query:
select * from table where column = 10;
could skip reading the whole row group if the dictionary for column has values [5, 11, 17, 20]
This could save IO and improve performance.

We implemented predicate pushdown using dictionary in Presto for parquet files, and benchmark shows up to 40X speedup for selective queries.

Need to add an api to ParquetFileReader, so that it returns dictionaries for requested columns.
If the column is not dictionary encoded in this row group, return null.
If the not all column pages are dictionary encoded in this row group, return null.

Issue Links

• is superceded by: PARQUET-384 Add Dictionary Based Filtering to Filter2 API