Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-19345

In Table File Sink, introduce streaming sink compaction

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.12.0
    • Table SQL / Runtime
    • None

    Description

      Users often complain that many small files are written out. Small files will affect the performance of file reading and the DFS system, and even the stability of the DFS system.

      Target: 

      • Compact all files generated by this job in a single checkpoint.
      • With compaction, Users can have smaller checkpoint interval, even to seconds.

      Document: https://docs.google.com/document/d/1cdlyoqgBq9yJEiHFBziimIoKHapQiEY2-0Tn8IF6G-c/edit?usp=sharing

      Attachments

        Issue Links

          Activity

            People

              lzljs3620320 Jingsong Lee
              lzljs3620320 Jingsong Lee
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: