Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-1658 [UMBRELLA] Spark Sql Support For Hudi
  3. HUDI-2259

[SQL]Support referencing subquery with column aliases by table alias in merge into

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersStop watchingWatchersConvert to IssueLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

       
      Example:

      val tableName = "test_hudi_table"
      
      spark.sql(
      s"""
      create table ${tableName} (
      id int,
      name string,
      price double,
      ts long
      ) using hudi
      options (
      primaryKey = 'id',
      type = 'cow'
      )
      location '/tmp/${tableName}'
      """.stripMargin)
      
      spark.sql(
      s"""
      merge into $tableName as t0
      using (
      select 1, 'a1', 12, 1003
      ) s0 (id,name,price,ts)
      on s0.id = t0.id
      when matched and id != 1 then update set *
      when matched and s0.id = 1 then delete
      when not matched then insert *
      """.stripMargin)
      

      It will throw an exception:

      Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot resolve 's0.id in (`s0.id` = `t0.id`), the input columns is: id#4, name#5, price#6, ts#7, _hoodie_commit_time#8, _hoodie_commit_seqno#9, _hoodie_record_key#10, _hoodie_partition_path#11, _hoodie_file_name#12, id#13, name#14, price#15, ts#16L;Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot resolve 's0.id in (`s0.id` = `t0.id`), the input columns is: id#4, name#5, price#6, ts#7, _hoodie_commit_time#8, _hoodie_commit_seqno#9, _hoodie_record_key#10, _hoodie_partition_path#11, _hoodie_file_name#12, id#13, name#14, price#15, ts#16L; at org.apache.spark.sql.hudi.analysis.HoodieResolveReferences.org$apache$spark$sql$hudi$analysis$HoodieResolveReferences$$resolveExpressionFrom(HoodieAnalysis.scala:292) at org.apache.spark.sql.hudi.analysis.HoodieResolveReferences$$anonfun$apply$1.applyOrElse(HoodieAnalysis.scala:160) at org.apache.spark.sql.hudi.analysis.HoodieResolveReferences$$anonfun$apply$1.applyOrElse(HoodieAnalysis.scala:103) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1.apply(AnalysisHelper.scala:89) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1.apply(AnalysisHelper.scala:86) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:194) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.resolveOperatorsUp(AnalysisHelper.scala:86) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUp(LogicalPlan.scala:29)

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            dongkelun 董可伦 Assign to me
            dongkelun 董可伦
            Votes:
            0 Vote for this issue
            Watchers:
            2 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment