[SPARK-38717] Handle Hive's bucket spec case preserving behaviour

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.1, 3.4.0
• Component/s: SQL
• Assignee: Peter Toth
• Reporter: Peter Toth
• Created: 31/Mar/22 16:23
• Updated: 16/Oct/22 08:50
• Resolved: 26/Sep/22 05:10

Description

CREATE TABLE t(
 c STRING,
 B_C STRING
)
PARTITIONED BY (p_c STRING)
CLUSTERED BY (B_C) INTO 4 BUCKETS
STORED AS PARQUET

then

SELECT * FROM t

fails with:

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Bucket columns B_C is not part of the table columns ([FieldSchema(name:c, type:string, comment:null), FieldSchema(name:b_c, type:string, comment:null)]
	at org.apache.hadoop.hive.ql.metadata.Table.setBucketCols(Table.java:552)
	at org.apache.spark.sql.hive.client.HiveClientImpl$.toHiveTable(HiveClientImpl.scala:1098)
	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getPartitionsByFilter$1(HiveClientImpl.scala:764)
	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:294)
	at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:225)
	at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:224)
	at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:274)
	at org.apache.spark.sql.hive.client.HiveClientImpl.getPartitionsByFilter(HiveClientImpl.scala:763)
	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$listPartitionsByFilter$1(HiveExternalCatalog.scala:1287)
	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:101)
	... 110 more

 

Issue Links

• links to: [Github] Pull Request #36027 (peter-toth)
• links to: [Github] Pull Request #37982 (peter-toth)
• links to: [Github] Pull Request #37982 (peter-toth)