[FLINK-34255] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 2.0.0
• Component/s: Runtime / Checkpointing, Runtime / State Backends

Release Note

Currently all the options about state and checkpointing are reorganized and categorized by prefixes as listed below:

1. execution.checkpointing: all configurations associated with checkpointing and savepoint.
2. execution.state-recovery: all configurations pertinent to state recovery.
3. state.*: all configurations related to the state accessing.
    1. state.backend.*: specific options for individual state backends, such as RocksDB.
    2. state.changelog: configurations for the changelog, as outlined in FLIP-158, including the options for the "Durable Short-term Log" (DSTL).
    3. state.latency-track: configurations related to the latency tracking of state access.

At the meantime, all the original options scattered everywhere are annotated as @Deprecated

• Assignee: Zakelly Lan
• Reporter: Zakelly Lan
• Created: 29/Jan/24 05:47
• Updated: 12/Oct/24 02:15

Description

The FLIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560

 

Currently, the configuration options pertaining to checkpointing, recovery, and state management are primarily grouped under the following prefixes:

• state.backend.* : configurations related to state accessing and checkpointing, as well as specific options for individual state backends
• execution.checkpointing.* : configurations associated with checkpoint execution and recovery
• execution.savepoint.{*}: configurations for recovery from savepoint

In addition, there are several individual options such as state.checkpoint-storage and state.checkpoints.dir that fall outside of these prefixes. The current arrangement of these options, which span multiple modules, is somewhat haphazard and lacks a systematic structure. For example, the options under the {{CheckpointingOptions }}and {{ExecutionCheckpointingOptions }}are related and have no clear boundaries from the user's perspective, but there is no unified prefix for them. With the upcoming release of Flink 2.0, we have an excellent opportunity to overhaul and restructure the configurations related to checkpointing, recovery, and state management. This FLIP proposes to reorganize these settings, making it more coherent by module, which would significantly lower the barriers for understanding and reduce the development costs moving forward.

Issue Links

• mentioned in: Page Loading...
• mentioned in: Page Loading...

Sub-Tasks

• Introduce StateRecoveryOptions in flink-core: Resolved
• Move RestoreMode from flink-runtime to flink-core: Resolved
• Move all checkpoint-related options into CheckpointingOptions: Resolved
• Move ExternalizedCheckpointCleanup to flink-core: Resolved
• Move CheckpointingMode to flink-core: Resolved
• Rename options for latency tracking: Resolved
• Rename options for Generalized incremental checkpoints (changelog): Resolved
• Rename options for checkpointing: Resolved
• Improve the documentation of 'state.checkpoints.dir' and 'state.checkpoint-storage': Closed
• Split 'state.backend.local-recovery' into two options for checkpointing and recovery: Resolved
• Provide compatibility of old CheckpointMode for connector testing framework: Closed
• Flink 2.0: Rename RestoreMode to RecoveryClaimMode: Resolved
• Flink 2.0: Remove legacy State&Checkpointing&Recovery options: Resolved
• Remove legacy ExternalizedCheckpointCleanup: Resolved