[FLINK-23139] State ownership: track and discard private state (registry+changelog)

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Fix Version/s: 1.16.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available stale-assigned
• Assignee: Unassigned
• Reporter: Roman Khachatryan
• Created: 24/Jun/21 07:54
• Updated: 15/Aug/22 14:26
• Resolved: 15/Aug/22 14:25

Description

TM should own changelog backend state to prevent re-uploading state on checkpoint abortion (or missing confirmation). A simpler solution to only own aborted state is less maintanable in the long run.

For that, on TM state should be tracked and discarded (on subsumption+materialization; on shutdown). 

See state ownership design doc, in particular Tracking private state.

 

This ticket is about creating TaskStateRegistry and using it in ChangelogStateBackend (for non-materialized part only; for materialized see FLINK-23344).
 

Externalized checkpoints and savepoints should be supported (or please create a separate ticket).

 

Retained checkpoints is a separate ticket: FLINK-23251

Issue Links

• blocks: FLINK-23343 State ownership: prevent shared state deletion on TM
• blocks: FLINK-23344 State ownership: track and discard private state (materialized: rocks, heap)
• is blocked by: FLINK-21504 State ownership: notify TMs about checkpoint subsumption
• is blocked by: FLINK-21353 Add FS-based StateChangelog implementation
• is blocked by: FLINK-21356 Implement incremental checkpointing and recovery using state changelog
• is blocked by: FLINK-21357 Add periodic materialization
• is cloned by: FLINK-23344 State ownership: track and discard private state (materialized: rocks, heap)
• relates to: FLINK-25511 Pre-emptively uploaded changelog not discarded up if materialized before checkpoint
• links to: GitHub Pull Request #16575