[SPARK-44058] Remove deprecated API usage in HiveShim.scala

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 3.4.0
• Component/s: Spark Submit
• Assignee: Unassigned
• Reporter: Aman Raj
• Created: 14/Jun/23 18:05
• Updated: 11/Jul/23 18:35
• Resolved: 11/Jul/23 18:35

Description

Spark's HiveShim.scala calls this particular method in Hive :
createPartitionMethod.invoke(
hive,
table,
spec,
location,
params, // partParams
null, // inputFormat
null, // outputFormat
-1: JInteger, // numBuckets
null, // cols
null, // serializationLib
null, // serdeParams
null, // bucketCols
null) // sortCols
}
 
We do not have any such implementation of createPartition in Hive. We only have this definition :
public Partition createPartition(Table tbl, Map<String, String> partSpec) throws HiveException {
    try

{       org.apache.hadoop.hive.metastore.api.Partition part =           Partition.createMetaPartitionObject(tbl, partSpec, null);       AcidUtils.TableSnapshot tableSnapshot = AcidUtils.getTableSnapshot(conf, tbl);       part.setWriteId(tableSnapshot != null ? tableSnapshot.getWriteId() : 0);       return new Partition(tbl, getMSC().add_partition(part));     }

catch (Exception e)

{       LOG.error(StringUtils.stringifyException(e));       throw new HiveException(e);     }

  }
The 12 parameter implementation was removed in HIVE-5951

 

The issue is that this 12 parameter implementation of createPartition method was added in Hive-0.12 and then was removed in Hive-0.13. When Hive 0.12 was used in Spark, SPARK-15334 commit in Spark added this 12 parameters implementation. But after Hive migrated to newer APIs somehow this was not changed in Spark OSS and it looks to us like a Bug from the Spark end.

 

We need to migrate to the newest implementation of Hive createPartition method otherwise this flow can break

Issue Links

• links to: [Github] Pull Request #41602 (amanraj2520)