[FLINK-33892] FLIP-383: Support Job Recovery from JobMaster Failures for Batch Jobs

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Junrui Li
• Reporter: Lijie Wang
• Created: 20/Dec/23 05:42
• Updated: 4 hours ago

Description

This is the umbrella ticket for FLIP-383

Issue Links

• links to: GitHub Pull Request #24771
• mentioned in: Page Loading...

Sub-Tasks

• Compute the number of subpartitions when initializing executon job vertices: Closed
• Introduce new config options for Batch Job Recovery: Closed
• Introduce JobEvent and JobEventStore for Batch Job Recovery: In Progress
• Introduce SupportsBatchSnapshot for operator coordinator: Closed
• Support obtain all partitions existing in cluster through ShuffleMaster.: Closed
• Extend shuffleMaster to support batch snapshot.: Closed
• TaskExecutor supports retain partitions after JM crashed.: Closed
• Support recover shuffle descriptor and partition metrics from tiered storage: Open