Details
-
New Feature
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Users often complain that many small files are written out. Small files will affect the performance of file reading and the DFS system, and even the stability of the DFS system.
Target:
- Compact all files generated by this job in a single checkpoint.
- With compaction, Users can have smaller checkpoint interval, even to seconds.
Document: https://docs.google.com/document/d/1cdlyoqgBq9yJEiHFBziimIoKHapQiEY2-0Tn8IF6G-c/edit?usp=sharing
Attachments
Issue Links
- is blocked by
-
FLINK-19356 Introduce FileLifeCycleListener to Buckets
- Closed
-
FLINK-19357 Introduce createBucketWriter to BucketsBuilder
- Closed
-
FLINK-19707 Refactor table streaming file sink
- Closed
- is related to
-
FLINK-17505 Merge small files produced by StreamingFileSink
- Closed
- mentioned in
-
Page Loading...