[IMPALA-1480] Slow DDL statements for tables with large number of partitions

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.0
• Fix Version/s: Impala 2.5.0
• Component/s: Catalog
• Labels: impala performance
• Target Version: Impala 2.5.0
• Assignee: Dimitris Tsirogiannis
• Reporter: Dimitris Tsirogiannis
• Created: 18/Nov/14 21:54
• Updated: 24/May/18 17:46
• Resolved: 28/Jan/16 21:26

Description

Impala users sometimes report that DDL statements (e.g. alter table partition set location...) are taking multiple seconds (>5) for partitioned tables with large number of partitions. The same operations are significantly faster in hive (sub-second response time).

Use case:

• 2 node cluster
• Single table (24 columns, 3 partition keys) with 2500 partitions
• alter table foo partition (foo_i = i) set location 'hdfs://.....' takes approximately 5-6sec (0.2 in HIVE)
• 1 sec delay in the alter stmt is caused by https://issues.apache.org/jira/browse/HIVE-5524