Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-6483

MERGE INTO should support schema evolution for partial updates.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • 0.16.0, 1.1.0
    • spark-sql
    • None

    Description

      Following code is example for doing MERGE INTO along with schema evolution which is not yet supported by hudi. Currently, Hudi tries to use target table schema during MERGE INTO.

      Following code should be supported - 

      ```
      create table test_insert3 (
          id int,
      name string,
      updated_at timestamp
      ) using hudi
      options (
          type = 'cow',
          primaryKey = 'id',
          preCombineField = 'updated_at'
      ) location 'file:///tmp/test_insert3';

      merge into test_insert3 as target
      using (
          select 1 as id, 'c' as name, 1 as new_col, current_timestamp as updated_at
      ) source
      on target.id = source.id
      when matched then update set target.new_col = source.new_col
      when not matched then insert *;
      ```

      Attachments

        Activity

          People

            Unassigned Unassigned
            adityagoenka Aditya Goenka
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: