Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-2706

Refactor Spark-SQL to make consistent with DataFrame

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Blocker
    • Resolution: Unresolved
    • None
    • 0.10.0
    • spark

    Description

      1. refactor `CreateHoodieTableCommand`;
        1. use `TBLPROPERTIES` syntax to pass config rather than `OPTIONS`, and keep compatible with `OPTIONS` syntax;
        2. force to provide `PrimaryKey`, so that make Update/Delete available;
      2. hudi spark-sql decouple from metastore as far as possible, get config from local hoodie.properties first;
      3. modify operation when insert/merge, make consistent with dataframe api.
      4. add parameter validation for recordKey, preCombineKey, keyGenerator, even if those parameters are defined by different keys.

      Attachments

        Issue Links

          Activity

            People

              biyan900116@gmail.com Yann Byron
              biyan900116@gmail.com Yann Byron
              Raymond Xu, Shaofeng Li
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: