[HUDI-1557] Make Flink write pipeline write task scalable

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 0.8.0
• Component/s: flink
• Labels: pull-request-available
• Assignee: Danny Chen
• Reporter: Danny Chen
• Created: 28/Jan/21 12:22
• Updated: 07/Feb/21 02:29
• Resolved: 06/Feb/21 14:04

Description

This issue introduces a BucketAssigner that assigns bucket ID (partition path & fileID) for each stream record.

There is no need to look up index and partition the records anymore in the following pipeline for these records,
we actually decide the write target location before the write and each record computes its location when the BucketAssigner receives it, thus, the indexing is with streaming style.

Computing locations for a batch of records all at a time is resource consuming so a pressure to the engine,
we should avoid that in streaming system.

Issue Links

• links to: GitHub Pull Request #2506