[SPARK-42480] Improve the performance of drop partitions

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.2
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: Wechar
• Reporter: Wechar
• Created: 17/Feb/23 21:10
• Updated: 30/Aug/23 11:49
• Resolved: 09/Mar/23 03:30

Description

Currently to drop the matching partitions, Spark will first get all matching Partition objects from Hive metastore, and just use the partition values of these Partition objects.

We can get the matching partition names instead of the partition objects for the following reasons:
1. we can also get partition values through a partition name (like a=1/b=2)
2. the byte size of partition name is much smaller than partition object, which will help improve the performance of drop partitions.

Issue Links

• supercedes: SPARK-33628 Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl
• links to: [Github] Pull Request #40069 (wecharyu)