Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-25842 [v2] FLIP-158: Generalized incremental checkpoints
  3. FLINK-23139

State ownership: track and discard private state (registry+changelog)

    XMLWordPrintableJSON

Details

    Description

      TM should own changelog backend state to prevent re-uploading state on checkpoint abortion (or missing confirmation). A simpler solution to only own aborted state is less maintanable in the long run.

      For that, on TM state should be tracked and discarded (on subsumption+materialization; on shutdown). 

      See state ownership design doc, in particular Tracking private state.

       

      This ticket is about creating TaskStateRegistry and using it in ChangelogStateBackend (for non-materialized part only; for materialized see FLINK-23344).
       

      Externalized checkpoints and savepoints should be supported (or please create a separate ticket).

       

      Retained checkpoints is a separate ticket: FLINK-23251

      Attachments

        Issue Links

          blocks

          Sub-task - The sub-task of the issue FLINK-23343 State ownership: prevent shared state deletion on TM

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue FLINK-23344 State ownership: track and discard private state (materialized: rocks, heap)

          • Major - Major loss of function.
          • Closed
          is blocked by

          Sub-task - The sub-task of the issue FLINK-21504 State ownership: notify TMs about checkpoint subsumption

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue FLINK-21353 Add FS-based StateChangelog implementation

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue FLINK-21356 Implement incremental checkpointing and recovery using state changelog

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue FLINK-21357 Add periodic materialization

          • Major - Major loss of function.
          • Closed
          is cloned by

          Sub-task - The sub-task of the issue FLINK-23344 State ownership: track and discard private state (materialized: rocks, heap)

          • Major - Major loss of function.
          • Closed
          relates to

          Sub-task - The sub-task of the issue FLINK-25511 Pre-emptively uploaded changelog not discarded up if materialized before checkpoint

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #16575

          Activity

            People

              Unassigned Unassigned
              roman Roman Khachatryan
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: