[IMPALA-4762] RECOVER PARTITIONS should send new partitions in small batches to HMS

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.5.0
• Fix Version/s: Impala 2.9.0
• Component/s: Catalog
• Labels: ramp-up
• Target Version: Product Backlog
• Assignee: Zach Amsden
• Reporter: Juan Yu
• Created: 12/Jan/17 18:47
• Updated: 15/Jun/17 17:15
• Resolved: 22/Mar/17 02:41

Description

HMS cannot handle more than 32k partitions in one call. When adding large amount of partitions, Impala should send the new partitions in smaller batches. otherwise HMS could go OOM.

Issue Links

• relates to: IMPALA-880 COMPUTE STATS should update partitions in batches