Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-8321

SQL insert mode needs to set merger as well to support non avro records

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • 1.1.0
    • spark, spark-sql, writer-core
    • None

    Description

      org.apache.spark.sql.hudi.ProvidesHoodieConfig deduces the payload to sometimes be ValidateDuplicateKeyPayload. This is only relevant for avro merging, and the validation will not happen if spark records are used. 
      HoodieSparkValidateDuplicateKeyRecordMerger should also be set here so that validation happens

      Attachments

        Issue Links

          is a child of

          Bug - A problem which impairs or prevents the functions of the product. HUDI-8317 Spark Record / Spark Merger gaps with Avro during writing

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Activity

            People

              Unassigned Unassigned
              jonvex Jonathan Vexler
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: