[HUDI-3446] Support Batch Reader in BootstrapOperator#loadRecords

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Component/s: flink
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Bo Cui
• Created: 17/Feb/22 10:28
• Updated: 22/Feb/22 13:46
• Resolved: 22/Feb/22 13:46

Description

https://github.com/apache/hudi/blob/433c2573ef5d1f25cba42f169679a4afe5ff2e24/hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/BootstrapOperator.java#L216

In our production environment, restarting the flink job requires a lot of memory to load records and send to the StateBackend of BucketAssignFunction. because before sending records, all records needs to be cached in the heap of TM.

Issue Links

• links to: GitHub Pull Request #4837