[IMPALA-3128] Parallelize file/block metadata loading for tables with large number of partitions

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.2.4
• Fix Version/s: Impala 2.10.0
• Component/s: Catalog
• Labels: catalog-server performance
• Target Version: Product Backlog
• Assignee: Bharath Vissapragada
• Reporter: Dimitris Tsirogiannis
• Created: 03/Mar/16 02:16
• Updated: 13/Dec/17 08:00
• Resolved: 13/Dec/17 08:00

Description

Currently, loading file/block metadata of a table is performed by a single thread. For large tables, loading metadata may take a long time, thereby increasing the latency during planning and/or other DDL/DML operations. We should consider parallelizing this process.