[FLINK-25555] FLIP-191: Extend unified Sink interface to support small file compaction

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.15.0
• Component/s: API / Core, API / DataStream, Connectors / Common
• Assignee: Fabian Paul
• Reporter: Fabian Paul
• Created: 06/Jan/22 09:13
• Updated: 22/Mar/22 12:34
• Resolved: 22/Mar/22 12:34

Description

Introduce Sink V2 offering unified exactly-once processing and custom extension hooks to better support data lakes.

 

More details see: https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction

Issue Links

• supercedes: FLINK-17505 Merge small files produced by StreamingFileSink

Sub-Tasks

• Introduce decomposed Sink V2 interfaces: Closed
• Introduce Sink V2 extension interfaces: Closed
• Implement StreamGraph translation for Sink V2 interfaces: Closed
• Update Elasticsearch Sink to use decomposed interfaces: Closed
• Update File Sink to use decomposed interfaces: Closed
• Update Kafka Sink to use decomposed interfaces: Closed
• Update Async Sink to use decomposed interfaces: Closed
• Graduate Sink V1 interfaces to PublicEvolving and deprecate them: Closed
• Support compacting small files for FileSink: Closed
• Implement GlobalCommitter as custom post commit topology: Closed
• Update docs to introduce the compaction for FileSink: Closed
• Test FileSink compaction manually: Closed