[HUDI-1624] The state based index should bootstrap from existing base files

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Component/s: flink
• Labels: pull-request-available
• Assignee: Danny Chen
• Reporter: Danny Chen
• Created: 18/Feb/21 05:44
• Updated: 24/Feb/21 01:54
• Resolved: 23/Feb/21 05:38

Description

The index should bootstrap from existing base files if there are, in the design, we load all the keys for one partition if we found that the key does not exist in the index for processElement, if there are many records for this partition, the processing may block and trigger back pressure. When all the records are loaded, we only need to check the state each time a record is tagged.

Issue Links

• links to: GitHub Pull Request #2581