[FLINK-25817] FLIP-201: Persist local state in working directory

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.15.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Till Rohrmann
• Reporter: Till Rohrmann
• Created: 26/Jan/22 09:38
• Updated: 21/Feb/22 08:17
• Resolved: 08/Feb/22 14:21

Description

This issue is the umbrella ticket for FLIP-201 which aims at adding support for persisting local state in Flink's working directory. This would enable Flink in certain scenarios to recover locally even in case of process failures.

Issue Links

• blocks: FLINK-25865 Support to set restart policy of TaskManager pod for native K8s integration
• relates to: FLINK-25831 ExecutionVertex.getLatestPriorAllocation fails if there is an unsuccessful restart attempt
• relates to: FLINK-26003 Use Jackson serialization for persisting TaskExecutor state to working directory
• relates to: FLINK-26014 Document how to use the working directory for faster local recoveries
• relates to: FLINK-26274 Test local recovery works across TaskManager process restarts
• links to: Draft PR
• links to: FLIP-201
• links to: GitHub Pull Request #18237

Sub-Tasks

• ExecutionVertex.getLatestPriorAllocation fails if there is an unsuccessful restart attempt: Closed
• Differentiate TaskManager sessions when registering at the JobMaster: Closed
• DefaultDeclarativeSlotPool rejects offered slots when the job is restarting: Closed