Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-19345

In Table File Sink, introduce streaming sink compaction

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.12.0
    • Component/s: Table SQL / Runtime
    • Labels:
      None

      Description

      Users often complain that many small files are written out. Small files will affect the performance of file reading and the DFS system, and even the stability of the DFS system.

      Target: 

      • Compact all files generated by this job in a single checkpoint.
      • With compaction, Users can have smaller checkpoint interval, even to seconds.

      Document: https://docs.google.com/document/d/1cdlyoqgBq9yJEiHFBziimIoKHapQiEY2-0Tn8IF6G-c/edit?usp=sharing

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                lzljs3620320 Jingsong Lee
                Reporter:
                lzljs3620320 Jingsong Lee
              • Votes:
                1 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: