Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      This issue is the very first step towards incremental checkpointing. We introduce a new state handle named CompositeStateHandle to be the base of the snapshots taken by task components. Known implementation may include KeyedStateHandle (for KeyedStateBackend}}s), {{SubtaskState (for subtasks, splits of JobVertex) and TaskState (for {{JobVertex}}s).

      Each CompositeStateHandle is composed of a collection of StateObject}s. It should register all its state objects in {{StateRegistry when its checkpoint is added into CompletedCheckpointStore (i.e., a pending checkpoint completes or a complete checkpoint is reloaded in the recovery).

      When a completed checkpoint is moved out of the CompletedCheckpointStore, we should not simply discard all state objects in the checkpoint. With the introduction of incremental checkpointing, a StateObject may be referenced by different checkpoints. We should unregister all the state objects contained in the StateRegistry first. Only those state objects that are not referenced by any checkpoint can be deleted.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                xiaogang.shi Xiaogang Shi
                Reporter:
                xiaogang.shi Xiaogang Shi
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: