[IMPALA-4625] Parquet Row Group Size optimization

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Do
• Affects Version/s: Impala 2.2.4
• Component/s: Backend
• Epic Link: selective query optimizations
• Assignee: Unassigned
• Reporter: Alan Choi
• Created: 08/Dec/16 01:41
• Updated: 04/Oct/18 17:45
• Resolved: 04/Oct/18 17:45

Description

For highly selective query, once we can narrow it down to a single Parquet file that matches the predicate, Impala still has to process that single file. Right now, a single Parquet file generated by Impala has only 1 row group and its default size is 256MB. A row group is processed in serial cannot be parallelized at this moment. Processing a 256MB row group can takes several sec, even for a very simple predicate. To improve the response time of such highly selective query, the row group must be smaller.

Issue Links

• is related to: IMPALA-5843 Use page index in Parquet files to skip pages