[HIVE-27505] Added batching with retry mechanism in Hive.getAllPartitionsOf API

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-beta-1
• Labels: pull-request-available
• Assignee: Vikram Ahuja
• Reporter: Vikram Ahuja
• Created: 14/Jul/23 09:37
• Updated: 15/Aug/23 06:33
• Resolved: 29/Jul/23 16:30

Description

Hive.getAllPartitionsOf() tries to list partitions in one go, it can cause memory related issues with wide tables(>2000 columns) with large amount of partitions when data is being transferred from HMS to HS2 using Thrift. Optimise the flow to use PartitionIterable so that partitions can be fetched in batches rather than one go.

Issue Links

• links to: GitHub Pull Request #4501