Details
-
Task
-
Status: Closed
-
Major
-
Resolution: Done
-
None
Description
Example:
val tableName = "test_hudi_table" spark.sql( s""" create table ${tableName} ( id int, name string, price double, ts long ) using hudi options ( primaryKey = 'id', type = 'cow' ) location '/tmp/${tableName}' """.stripMargin) spark.sql( s""" merge into $tableName as t0 using ( select 1, 'a1', 12, 1003 ) s0 (id,name,price,ts) on s0.id = t0.id when matched and id != 1 then update set * when matched and s0.id = 1 then delete when not matched then insert * """.stripMargin)
It will throw an exception:
Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot resolve 's0.id in (`s0.id` = `t0.id`), the input columns is: id#4, name#5, price#6, ts#7, _hoodie_commit_time#8, _hoodie_commit_seqno#9, _hoodie_record_key#10, _hoodie_partition_path#11, _hoodie_file_name#12, id#13, name#14, price#15, ts#16L;Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot resolve 's0.id in (`s0.id` = `t0.id`), the input columns is: id#4, name#5, price#6, ts#7, _hoodie_commit_time#8, _hoodie_commit_seqno#9, _hoodie_record_key#10, _hoodie_partition_path#11, _hoodie_file_name#12, id#13, name#14, price#15, ts#16L; at org.apache.spark.sql.hudi.analysis.HoodieResolveReferences.org$apache$spark$sql$hudi$analysis$HoodieResolveReferences$$resolveExpressionFrom(HoodieAnalysis.scala:292) at org.apache.spark.sql.hudi.analysis.HoodieResolveReferences$$anonfun$apply$1.applyOrElse(HoodieAnalysis.scala:160) at org.apache.spark.sql.hudi.analysis.HoodieResolveReferences$$anonfun$apply$1.applyOrElse(HoodieAnalysis.scala:103) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1.apply(AnalysisHelper.scala:89) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1.apply(AnalysisHelper.scala:86) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:194) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.resolveOperatorsUp(AnalysisHelper.scala:86) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUp(LogicalPlan.scala:29)