Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-1628

Improve data locality during ingestion

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Writer Core
    • Labels:

      Description

      Today the upsert partitioner does the file sizing/bin-packing etc for
      inserts and then sends some inserts over to existing file groups to
      maintain file size.
      We can abstract all of this into strategies and some kind of pipeline
      abstractions and have it also consider "affinity" to an existing file group
      based
      on say information stored in the metadata table?

      See http://mail-archives.apache.org/mod_mbox/hudi-dev/202102.mbox/browser
      for more details

        Attachments

          Activity

            People

            • Assignee:
              thirumalai.raj Thirumalai Raj R
              Reporter:
              satishkotha satish
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: