[HUDI-1521] [UMBRELLA] RFC-24 HUDI Flink writer proposal

• Type: Epic
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 0.8.0
• Component/s: flink
• Assignee: Danny Chen
• Reporter: Danny Chen
• Created: 11/Jan/21 07:18
• Updated: 09/May/22 09:21
• Resolved: 24/Mar/21 14:07

Description

As the RFC-24 has described [1], we would promote the Flink writer as following:

1. Remove the single parallelism operator and add test framework
2. Make the write task scalable
3. Write as mini-batch
4. Add a new index

So this is an umbrella issue, we would fix each as sub-tasks.

[1] https://cwiki.apache.org/confluence/display/HUDI/RFC+-+24%3A+Hoodie+Flink+Writer+Proposal

Issue Links

• relates to: HUDI-184 Integrate Hudi with Apache Flink

Sub-Tasks

• Add a new pipeline for Flink writer: Closed
• Make Flink write pipeline write task scalable: Closed
• Refactor the BaseFlinkCommitActionExecutor to adapt to the new Flink writer: Closed
• Write as minor batches during one checkpoint interval for the new writer: Closed
• The state based index should bootstrap from existing base files: Closed
• Supports merge on read write mode for Flink writer: Closed
• Avoid to rename for bucket update when there is only one flush action during a checkpoint: Closed
• Some improvements to BucketAssignFunction: Closed
• Supports snapshot read for Flink: Closed
• Streaming read for Flink MOR table: Closed
• Avro schema inference for Flink SQL table: Closed
• Row level delete for Flink sink: Closed
• Support object storage for Flink writer: Closed
• Tweak hudi-flink-bundle module pom and re-organize the pacakges for hudi-flink module: Closed
• Bounded source for stream writer: Closed
• Add document about HUDI Flink integration: Closed
• Implement HoodieTableSource.explainSource for all kinds of pushing down: Closed
• Use PRIMARY KEY syntax to define record keys for Flink Hudi table: Closed
• Flush as per data bucket for mini-batch write: Closed
• Read optimized query type for Flink batch reader: Closed