[SPARK-45054] HiveExternalCatalog.listPartitions should restore Spark SQL stats

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.4, 3.3.2, 3.4.1
• Fix Version/s: 3.4.2, 3.5.0, 4.0.0
• Component/s: SQL
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 01/Sep/23 18:14
• Updated: 02/Sep/23 03:25
• Resolved: 02/Sep/23 03:22

Description

If partitions are stored in HMS with Spark populated stats such as spark.sql.statistics.totalSize, currently HiveExternalCatalog.listPartitions doesn't call restorePartitionMetadata to restore those stats.