[HUDI-2426] spark sql extensions breaks read.table from metastore

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Won't Fix
• Fix Version/s: 0.10.1
• Component/s: spark
• Labels: sev:critical user-support-issues
• Story Points: 1
• Assignee: Yann Byron
• Reporter: nicolas paris

Reviewers

Shiyan Xu

• Created: 13/Sep/21 13:26
• Updated: 11/Jan/22 01:19
• Resolved: 11/Jan/22 01:19

Description

when adding the hudi spark sql support, this breaks the ability to read a hudi metastore from spark:

 bash-4.2$ ./spark3.0.2/bin/spark-shell --packages org.apache.hudi:hudi-spark3-bundle_2.12:0.9.0,org.apache.spark:spark-avro_2.12:3.1.2 --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension'

 

scala> spark.table("default.test_hudi_table").show
java.lang.UnsupportedOperationException: Unsupported parseMultipartIdentifier method
at org.apache.spark.sql.parser.HoodieCommonSqlParser.parseMultipartIdentifier(HoodieCommonSqlParser.scala:65)
at org.apache.spark.sql.SparkSession.table(SparkSession.scala:581)
... 47 elided

 

removing the config makes the hive table readable again from spark

this affect at least spark 3.0.x and 3.1.x

Issue Links

• is fixed by: HUDI-3131 Spark3.1.1 CTAS error
• relates to: HUDI-1869 Upgrading Spark3 To 3.1