Details
-
New Feature
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Users often complain that many small files are written out. Small files will affect the performance of file reading and the DFS system, and even the stability of the DFS system.
Target:
- Compact all files generated by this job in a single checkpoint.
- With compaction, Users can have smaller checkpoint interval, even to seconds.
Document: https://docs.google.com/document/d/1cdlyoqgBq9yJEiHFBziimIoKHapQiEY2-0Tn8IF6G-c/edit?usp=sharing
Attachments
Issue Links
- is blocked by
-
FLINK-19356 Introduce FileLifeCycleListener to Buckets
- Closed
-
FLINK-19357 Introduce createBucketWriter to BucketsBuilder
- Closed
-
FLINK-19707 Refactor table streaming file sink
- Closed
- is related to
-
FLINK-17505 Merge small files produced by StreamingFileSink
- Closed
- mentioned in
-
Page Loading...
Activity
Field | Original Value | New Value |
---|---|---|
Link |
This issue is related to |
Link |
This issue is blocked by |
Link |
This issue is blocked by |
Link |
This issue is blocked by |
Remote Link | This issue links to "Page (Apache Software Foundation)" [ 219281 ] |
Resolution | Fixed [ 1 ] | |
Status | Open [ 1 ] | Closed [ 6 ] |
Summary | Introduce File streaming sink compaction | In Table File Sink, introduce streaming sink compaction |
Remote Link | This issue links to "Page (Apache Software Foundation)" [ 226324 ] |
Remote Link | This issue links to "Page (Apache Software Foundation)" [ 226324 ] |
Hi aljoscha pnowojski kkl0u What do you think? And related
FLINK-19356FLINK-19357.CC: gaoyunhaii maguowei