Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-11499

Extend StreamingFileSink BulkFormats to support arbitrary roll policies

    XMLWordPrintableJSON

Details

    Description

      Currently when using the StreamingFilleSink Bulk-encoding formats can only be combined with the `OnCheckpointRollingPolicy`, which rolls the in-progress part file on every checkpoint.

      However, many bulk formats such as parquet are most efficient when written as large files; this is not possible when frequent checkpointing is enabled. Currently the only work-around is to have long checkpoint intervals which is not ideal.

       

      The StreamingFileSink should be enhanced to support arbitrary roll policy's so users may write large bulk files while retaining frequent checkpoints.

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue FLINK-10114 Support Orc for StreamingFileSink

          • Major - Major loss of function.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-17505 Merge small files produced by StreamingFileSink

          • Not a Priority -
          • Closed

          Activity

            People

              Unassigned Unassigned
              sjwiesman Seth Wiesman
              Votes:
              2 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

                Created:
                Updated: