[DRILL-4849] Refresh table metadata performance: read only new/updated parquet files

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.7.0
• Component/s: Query Planning & Optimization
• Labels: metadata parquet performance planning
• Assignee: Unassigned
• Reporter: Miroslav Holubec
• Created: 17/Aug/16 08:07
• Updated: 17/Aug/16 09:04

Description

Currently REFRESH TABLE METADATA takes serious amount of time for many small parquet files. We can actually only read these parquets, which are new or changed. This will require to add modificationTime into file metadata.