Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-1658 [UMBRELLA] Spark Sql Support For Hudi
  3. HUDI-2343

Fix the exception for mergeInto when the primaryKey and preCombineField of source table and target table differ in case only

    XMLWordPrintableJSON

Details

    Description

      Example:

      val tableName = "test_hudi_table"
      
      spark.sql(
      s"""
      create table ${tableName} (
      id int,
      name string,
      price double,
      ts long
      ) using hudi
      options (
      primaryKey = 'id',
      type = 'cow'
      )
      location '/tmp/${tableName}'
      """.stripMargin)
      
      spark.sql(
        s"""
           | merge into $tableName
           | using (
           |  select 1 as ID, 'a1' as NAME, 10 as PRICE, 1000 as TS, '1' as FLAG
           | ) s0
           | on s0.ID = $tableName.id
           | when matched and FLAG = '1' then update set
           | id = s0.ID, name = s0.NAME, price = s0.PRICE, ts = s0.TS
           | when not matched and FLAG = '1' then insert *
       """.stripMargin)

      It will throw an exception:

      org.apache.hudi.exception.HoodieException: ts(Part -ts) field not found in record. Acceptable fields were :[ID, NAME, PRICE, TS, FLAG]
      

      and:

      spark.sql(
        s"""
           | merge into $tableName
           | using (
           |  select 1 as ID, 'a1' as NAME, 10 as PRICE, 1000 as ts, '1' as FLAG
           | ) s0
           | on s0.ID = $tableName.id
           | when matched and FLAG = '1' then update set
           | id = s0.ID, name = s0.NAME, price = s0.PRICE, ts = s0.ts
           | when not matched and FLAG = '1' then insert *
       """.stripMargin)
      

      It will throw the exception:

      org.apache.hudi.exception.HoodieKeyException: recordKey values: "id:__null__" for fields: [id] cannot be entirely null or empty.
      

       

       

       

      Attachments

        Issue Links

          Activity

            People

              dongkelun 董可伦
              dongkelun 董可伦
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: