[FLINK-17589] Extend StreamingFileSink to Support Streaming Hive Connector

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Fix Version/s: 1.20.0
• Component/s: Connectors / FileSystem
• Labels: auto-deprioritized-major auto-unassigned
• Assignee: Unassigned
• Reporter: Yun Gao
• Created: 09/May/20 04:14
• Updated: 11/Mar/24 12:44

Description

According to the discussion on FLIP-115 , we will use StreamingFileSink to support the streaming hive connector. This requires to make some extension to the current StreamingFileSink:

1.   Support path-based writer to write the specified Hadoop path.
2.   Provides listeners to collect the bucket state so that the Hive sink could decide when one bucket is terminated and then writing meta-info to Hive.

Issue Links

• blocks: FLINK-14255 Integrate hive to streaming file sink

Sub-Tasks

• Add Bucket lifecycle listener to support acquiring bucket state: Closed
• Support arbitrary recovery mechanism for PartFileWriter: Closed
• Support Hadoop path-based PartFileWriter with renaming committer: Closed
• Support hadoop OutputCommitter based file committer: Reopened