[IMPALA-4524] In CatalogOpExecutor.alterTableAddPartitions() make RPC calls to HMS in batches

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0
• Component/s: Catalog, Frontend
• Labels: newbie performance ramp-up
• Target Version: Product Backlog
• Assignee: Philip Martin
• Reporter: Attila Jeges
• Created: 23/Nov/16 13:34
• Updated: 24/Oct/17 18:59
• Resolved: 24/Oct/17 18:59

Description

CatalogOpExecutor.alterTableAddPartitions() should adhere to MAX_PARTITION_UPDATES_PER_RPC limit when making RPC calls to the HMS. We've seen problems in the past when updating a lot of metadata in one RPC to the HMS.

Note that getPartitionsFromHms() and alterTableCachePartitions() methods called from alterTableAddPartitions() are also affected.

Currently it is enforced during analysis (in AlterTableAddPartitionStmt.analyze()) that one ALTER TABLE ADD PARTITION statement cannot add more than MAX_PARTITION_UPDATES_PER_RPC partitions. This check can be removed after fixing the RPC calls.