Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-9485 Improving Flink’s timer management for large state
  3. FLINK-9487

Prepare InternalTimerHeap for asynchronous snapshots

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Implemented
    • None
    • 1.6.0
    • None

    Description

      When we want to snapshot timers with the keyed backend state, this must happen as part of an asynchronous snapshot.

      The data structure InternalTimerHeap needs to offer support for this through a lightweight copy mechanism (e.g. arraycopy of the timer queue, because timers are immutable w.r.t. serialization).

      We can also stop keeping the dedup maps in InternalTimerHeap separated by key-group, all timers can go into one map.

      Instead, we can implement online-partitioning as part of the asynchronous operation, similar to what we do in CopyOnWriteStateTable snapshots. Notice that in this intermediate state, the code will still run in the synchronous part until we are integrated with the backends for async snapshotting (next subtask of this jira).

      Attachments

        Issue Links

          Activity

            People

              srichter Stefan Richter
              srichter Stefan Richter
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: