[DRILL-7028] Reduce the planning time of queries on large Parquet tables with large metadata cache files

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.17.0
• Component/s: Metadata
• Labels: performance
• Assignee: Venkata Jyothsna Donapati
• Reporter: Venkata Jyothsna Donapati
• Created: 06/Feb/19 00:05
• Updated: 14/Nov/19 17:27
• Resolved: 14/Nov/19 17:27

Description

If the Parquet table has a large number of small files, the metadata cache files grow larger and the planner tries to read the large metadata cache file which leads to the planning time overhead. Most of the time of execution is spent during the planning phase.

Issue Links

• relates to: DRILL-6552 Drill Metadata management "Drill Metastore"

Sub-Tasks

• Run-time row group pruning: Resolved
• Create separate summary file for schema, totalRowCount, totalNullCount (includes maintenance): Resolved
• Leverage the summary's totalRowCount and totalNullCount for COUNT() queries (also prevent eager expansion of files): Resolved
• Ensure backward compatibility is maintained: Resolved
• Auto-refresh should pick up existing columns from metadata cache: Resolved
• Refresh command to support subset of columns: Resolved
• Temp fix: Run-time rowgroup pruning match() fails on casting a Long to an Integer: Resolved
• Run-time rowgroup pruning match() fails on casting a Long to an Integer: Resolved