Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-21352

FLIP-158: Generalized incremental checkpoints

    XMLWordPrintableJSON

    Details

      Description

        Attachments

          Issue Links

          1.
          Add periodic materialization Sub-task In Progress Yuan Mei
          2.
          Write metadata after materialization Sub-task In Progress Yuan Mei
          3.
          State ownership: notify TMs about checkpoint subsumption Sub-task In Progress Yun Tang
          4.
          Prevent JM from discarding state on checkpoint abortion Sub-task Open Unassigned
          5.
          State ownership: compute shared state on recovery Sub-task Open Roman Khachatryan
          6.
          State ownership: track and discard private state (registry+changelog) Sub-task Open Roman Khachatryan
          7.
          State ownership: prevent private state deletion on JM Sub-task Open Zakelly Lan
          8.
          State ownership: track and discard private state (materialized: rocks, heap) Sub-task Open Unassigned
          9.
          State ownership: prevent shared state deletion on TM Sub-task Open Unassigned
          10.
          Provide backpressure (currently job fails if a limit is hit) Sub-task In Progress Roman Khachatryan
          11.
          Add a metric for back-pressure from the ChangelogStateBackend Sub-task Open Unassigned
          12.
          Add metrics for the ChangelogStateBackend Sub-task In Progress Roman Khachatryan
          13.
          Revert FLINK-23738 (i.e. unhide config, API, docs) Sub-task Closed Roman Khachatryan
          14.
          Add benchmarks for the ChangelogStateBackend Sub-task Open Unassigned
          15.
          Add explicit method to StateChangelogWriter to write metadata Sub-task Open Unassigned
          16.
          Add a benchmark for the Changelog DFS writer Sub-task Open Unassigned
          17.
          Hide any configuration, API or docs Sub-task Closed Roman Khachatryan
          18.
          Enable changelog backend in tests Sub-task Closed Roman Khachatryan
          19.
          Add percentiles to checkpoint stats Sub-task Closed Roman Khachatryan
          20.
          Cleanup unnecessary dependencies in dstl pom.xml Sub-task Closed Roman Khachatryan
          21.
          Optimize writing state changes Sub-task Closed Roman Khachatryan
          22.
          Add FS-based StateChangelog implementation Sub-task Closed Roman Khachatryan
          23.
          Add StateChangelog interface and its in-memory implementation Sub-task Closed Roman Khachatryan
          24.
          Provide StateBackends access to MailboxExecutor Sub-task Closed Yuan Mei
          25.
          Add ChangelogStateBackend (proxy-everything) Sub-task Closed Yuan Mei
          26.
          Send changes to the state changelog (still proxy everything) Sub-task Closed Roman Khachatryan
          27.
          Implement incremental checkpointing and recovery using state changelog Sub-task Closed Roman Khachatryan
          28.
          Test Changelog State backend (Wrapper) Sub-task Resolved Yangyang ZHANG
          29.
          Add Changelog state for timers (PQ) Sub-task Closed Piotr Nowojski
          30.
          Create and wire changelog writer with backend Sub-task Closed Zakelly Lan
          31.
          Log metadata Sub-task Closed Roman Khachatryan
          32.
          ChangelogStateBackend tests use nested backend on recovery Sub-task Closed Roman Khachatryan
          33.
          Adjust configuration of ChangelogStateBackend and hide it from users Sub-task Resolved Zakelly Lan
          34.
          Python API for enabling ChangelogStateBackend Sub-task Closed Zakelly Lan
          35.
          Consider disallowing in-memory state handles for materialized state Sub-task Open Unassigned
          36.
          Support recovery (from checkpoints) after disabling changelog backend Sub-task Open Unassigned
          37.
          Support state migration Sub-task Open Unassigned
          38.
          Support more than one retained checkpoints Sub-task Open Unassigned
          39.
          Expose some interfaces of changelog to users Sub-task Open Unassigned
          40.
          testKeyGroupedInternalPriorityQueue does not dispose ChangelogDelegateEmbeddedRocksDB properly, and fails the test Sub-task Closed Yun Tang
          41.
          Enable periodic materialisation in tests Sub-task In Progress Yuan Mei

            Activity

              People

              • Assignee:
                roman Roman Khachatryan
                Reporter:
                roman Roman Khachatryan
              • Votes:
                0 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated: