[HUDI-1598] Write as minor batches during one checkpoint interval for the new writer

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 0.8.0
• Component/s: flink
• Labels: pull-request-available
• Assignee: Danny Chen
• Reporter: Danny Chen
• Created: 08/Feb/21 07:28
• Updated: 23/Feb/21 13:10
• Resolved: 17/Feb/21 07:25

Description

Buffering data during one checkpoint when flush the buffer out all at a time is not resource friendly for streaming write. The more proper way it to cut the batches based on their real memory data buffer size (say, 128Mb), the writer always flushes the buffer out when its size reaches the configured threshold.

Thus, after this change, one instant may span one (if every checkpoint succeeds) or more (if there are checkpoint failures) checkpoints. The instant only commits when there is a successful checkpoint.

Issue Links

• links to: GitHub Pull Request #2553