[FLINK-19345] In Table File Sink, introduce streaming sink compaction

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.12.0
• Component/s: Table SQL / Runtime
• Assignee: Jingsong Lee
• Reporter: Jingsong Lee
• Created: 22/Sep/20 09:49
• Updated: 14/Jan/21 09:20
• Resolved: 11/Nov/20 03:59

Description

Users often complain that many small files are written out. Small files will affect the performance of file reading and the DFS system, and even the stability of the DFS system.

Target: 

• Compact all files generated by this job in a single checkpoint.
• With compaction, Users can have smaller checkpoint interval, even to seconds.

Document: https://docs.google.com/document/d/1cdlyoqgBq9yJEiHFBziimIoKHapQiEY2-0Tn8IF6G-c/edit?usp=sharing

Issue Links

• is blocked by: FLINK-19356 Introduce FileLifeCycleListener to Buckets
• is blocked by: FLINK-19357 Introduce createBucketWriter to BucketsBuilder
• is blocked by: FLINK-19707 Refactor table streaming file sink
• is related to: FLINK-17505 Merge small files produced by StreamingFileSink
• mentioned in: Page Loading...

Sub-Tasks

• Introduce File streaming compaction operators: Closed
• Introduce FileLifeCycleListener to Buckets: Closed
• Introduce createBucketWriter to BucketsBuilder: Closed
• Refactor table streaming file sink: Closed
• Integrate file compaction to filesystem connector: Closed
• Integrate file compaction to Hive connector: Closed
• Add document for file compaction: Closed