Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-1521 [UMBRELLA] RFC-24 HUDI Flink writer proposal
  3. HUDI-1624

The state based index should bootstrap from existing base files

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.8.0
    • flink

    Description

      The index should bootstrap from existing base files if there are, in the design, we load all the keys for one partition if we found that the key does not exist in the index for processElement, if there are many records for this partition, the processing may block and trigger back pressure. When all the records are loaded, we only need to check the state each time a record is tagged.

      Attachments

        Issue Links

          Activity

            People

              danny0405 Danny Chen
              danny0405 Danny Chen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: