[IMPALA-4986] Use Parquet statistics when evaluating min/max/count aggregates

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 2.9.0
• Component/s: Frontend
• Labels: parquet performance ramp-up
• Assignee: Unassigned
• Reporter: Lars Volker
• Created: 24/Feb/17 04:05
• Updated: 07/Sep/18 21:20

Description

There are various ways in which Parquet statistics such as num_rows and also parquet::Statistics can be used to speed up aggregation queries with min/max/count. Some of the improvements can be done at execution-time only, others also need query-plan modifications. The subtasks illustrate the various optimization opportunities/dimensions, and can be tackled separately.

Issue Links

• is related to: IMPALA-7547 For distinct queries use dictionary encoded page instead of reading all data

Sub-Tasks

• Use parquet::Statistics for simple min/max aggregates: Open
• Improve COUNT(*) performance of Parquet scans.: Resolved
• Use parquet::Statistics for min/max aggregates when only a subset of scan columns have stats: Open
• Apply Parquet stats optimizations in conjunction with predicates against Parquet stats: Open