[HUDI-3131] Spark3.1.1 CTAS error

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.1
• Component/s: spark
• Labels: pull-request-available
• Story Points: 1
• Assignee: Yann Byron
• Reporter: Yann Byron

Reviewers

Raymond Xu

• Created: 30/Dec/21 03:54
• Updated: 12/Jan/22 03:00
• Resolved: 11/Jan/22 01:20

Description

Fail to run CTAS with Hudi0.10.0 and Spark3.1.1 and Spark3.2.0.

For 3.1.1: 

Sql:

create table h1_p using hudi partitioned by(dt) options(type = 'cow', primaryKey = 'id') as select '2021-05-07' as dt, 1 as id, 'a1' as name, 10 as price; 

Error:

java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.plans.logical.Command.producedAttributes$(Lorg/apache/spark/sql/catalyst/plans/logical/Command;)Lorg/apache/spark/sql/catalyst/expressions/AttributeSet;
    at org.apache.spark.sql.hudi.command.CreateHoodieTableAsSelectCommand.producedAttributes(CreateHoodieTableAsSelectCommand.scala:39) 

 

For 3.2.0:

Sql:

create table hudi_tbl (id int, name string, price double, pt string) using hudi tblproperties (primaryKey = 'id') partitioned by (dt);
insert into hudi_tbl values(1, 'a1', 10, '2021-07-18');

-- error when exec this sql
create table hudi_ctas using hudi tblproperties( primaryKey = 'id' ) as select * from hudi_tbl;

Error:

java.lang.AssertionError: assertion failed: No plan for SubqueryAlias spark_catalog.default.h14
+- Relation default.h14[_hoodie_commit_time#4554,_hoodie_commit_seqno#4555,_hoodie_record_key#4556,_hoodie_partition_path#4557,_ho
  at scala.Predef$.assert(Predef.scala:223)
  at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:93)
  at org.apache.spark.sql.execution.SparkStrategies.plan(SparkStrategies.scala:68) 

Issue Links

• fixes: HUDI-2426 spark sql extensions breaks read.table from metastore
• is related to: HUDI-2811 Support Spark 3.2 and Parquet 1.12.x
• links to: GitHub Pull Request #4476
• links to: GitHub Pull Request #4549