Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-3993

Avoid calling into Spark UDF in Bulk Insert

    XMLWordPrintableJSON

Details

    Description

      Currently, invoking into UDF w/in Hudi's Bulk Insert causes  20% perf-gap as compared against raw Parquet Bulk Insert into a non-partitioned table.

       

      Attachments

        Issue Links

          fixes

          Task - A task that needs to be done. HUDI-4374 Support BULK_INSERT row-writing on streaming Dataset/DataFrame

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HUDI-4365 Bulk Insert not URL encoding Partition Path properly

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HUDI-3995 Bulk insert row writer perf improvements

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          Task - A task that needs to be done. HUDI-4036 Investigate whether meta fields could be omitted completely

          • Critical - Crashes, loss of data, severe memory leak.
          • Open
          links to

          Web Link GitHub Pull Request #5470

          Activity

            People

              alexey.kudinkin Alexey Kudinkin
              alexey.kudinkin Alexey Kudinkin
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: