[SPARK-32508] Disallow empty part col values in partition spec before static partition writing

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: SQL
• Assignee: dzcxzl
• Reporter: dzcxzl
• Created: 31/Jul/20 10:55
• Updated: 17/Sep/20 06:51
• Resolved: 17/Sep/20 06:51

Description

When writing to the current static partition, the partition field is empty, and an error will be reported when all tasks are completed.
We can prevent such behavior before submitting the task.

 

org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: get partition: Value for key d is null or empty;
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:113)
    at org.apache.spark.sql.hive.HiveExternalCatalog.getPartitionOption(HiveExternalCatalog.scala:1212)
    at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.getPartitionOption(ExternalCatalogWithListener.scala:240)
    at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:276)

 

 

Issue Links

• links to: [Github] Pull Request #29316 (cxzl25)