Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5053 Incremental / lightweight snapshots for checkpoints
  3. FLINK-6014

Allow the registration of state objects in checkpoints

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      This issue is the very first step towards incremental checkpointing. We introduce a new state handle named CompositeStateHandle to be the base of the snapshots taken by task components. Known implementation may include KeyedStateHandle (for KeyedStateBackend}}s), {{SubtaskState (for subtasks, splits of JobVertex) and TaskState (for {{JobVertex}}s).

      Each CompositeStateHandle is composed of a collection of StateObject}s. It should register all its state objects in {{StateRegistry when its checkpoint is added into CompletedCheckpointStore (i.e., a pending checkpoint completes or a complete checkpoint is reloaded in the recovery).

      When a completed checkpoint is moved out of the CompletedCheckpointStore, we should not simply discard all state objects in the checkpoint. With the introduction of incremental checkpointing, a StateObject may be referenced by different checkpoints. We should unregister all the state objects contained in the StateRegistry first. Only those state objects that are not referenced by any checkpoint can be deleted.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                shixg Xiaogang Shi
                Reporter:
                shixg Xiaogang Shi
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: