Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-1627

DeltaStreamer does not work with SQLtransformation

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None

      Description

      Hudi deltastreamer ,  using SQLtransformation :

      21/02/04 17:19:50 INFO SqlQueryBasedTransformer: SQL Query for transformation : (select order_date from `default`.ordertest)
      org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`ordertest`; line 1 pos 23;
      ’Project [’order_date]
      +- ’UnresolvedRelation `default`.`ordertest` 

       

      This is because:
      DeltaStreamer does not enable HiveSupport when initiating the SparkSession.
      this.sparkSession = SparkSession.builder().config(jssc.getConf()).getOrCreate();
      line 526 in https://github.com/a0x8o/hudi/blob/b8d0747959bc6f101b5b90b8e3ad323aafa2aa6e/hudi-u[…]rg/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              wenningd Wenning Ding
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: