Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-9485 Improving Flink’s timer management for large state
  3. FLINK-10026

Integrate heap based timers in RocksDB backend with incremental checkpoints

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Won't Do
    • 1.6.0
    • None
    • Runtime / Checkpointing
    • None

    Description

      Currently, heap based priority queue state does not participate in RocksDB incremental checkpointing and still uses the legacy (synchronous) way of checkpointing timers.

      We should integrate it with incremental checkpoints in such way that the state is written non-incrementally into a file, and another key-grouped state handle becomes part of the non-shared section of the incremental state handle.

      After that we can also remove the path for legacy checkpointing completely and also remove the key-grouping from the de-duplication hash map in the HeapPriorityQueueSet. All can go into one map.

      Attachments

        Activity

          People

            Unassigned Unassigned
            srichter Stefan Richter
            Votes:
            1 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: