[FLINK-4256] Fine-grained recovery

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.9.0
• Component/s: Runtime / Coordination
• Assignee: Stephan Ewen
• Reporter: Stephan Ewen
• Created: 22/Jul/16 16:37
• Updated: 02/Oct/19 17:45
• Resolved: 16/Aug/19 10:41

Description

When a task fails during execution, Flink currently resets the entire execution graph and triggers complete re-execution from the last completed checkpoint. This is more expensive than just re-executing the failed tasks.

In many cases, more fine-grained recovery is possible.

The full description and design is in the corresponding FLIP.

https://cwiki.apache.org/confluence/display/FLINK/FLIP-1+%3A+Fine+Grained+Recovery+from+Task+Failures

The detail desgin for version1 is https://docs.google.com/document/d/1_PqPLA1TJgjlqz8fqnVE3YSisYBDdFsrRX_URgRSj74/edit#

Issue Links

• is blocked by: FLINK-4322 Unify CheckpointCoordinator and SavepointCoordinator
• relates to: FLINK-13371 Release partitions in JM if producer restarts
• relates to: FLINK-12069 Add proper lifecycle management for intermediate result partitions
• relates to: FLINK-10288 Failover Strategy improvement
• links to: GitHub Pull Request #3539

Sub-Tasks

• Remove Serializabiliy of ExecutionGraph: Closed
• Fix misleading ScheduleMode names: Closed
• Initialize TaskExecutions directly with their starting state: Closed
• The implementation of FailoverRegion.: Closed
• The implementation of RestartPipelinedRegionStrategy: Closed
• Implement a new RestartStrategy that works for the FailoverRegion.: Closed
• ExecutionGraph use FailoverCoordinator to manage the failover of execution vertexes: Closed
• Introduce a TerminationFuture for Execution: Closed
• Introduce the abstract PartitionException for downstream task failure: Closed
• Batch Job: InputSplit Fault tolerant for DataSourceTask: Closed
• Backtrack failover regions if intermediate results are unavailable: Resolved
• Resetting ExecutionVertex in region failover may cause inconsistency of IntermediateResult status: Resolved
• Implement a region failover strategy based on new FailoverStrategy interfaces: Closed
• Introduce PartitionConnectionException for unreachable producer: Resolved
• Implement ExecutionGraph to FailoverTopology Adapter: Closed
• Add an adapter of region failover NG for legacy scheduler: Closed
• IT test for fine-grained recovery (user code failures): Closed
• Leverage JM side partition state to improve region failover experience: Closed
• IT test for fine-grained recovery (task executor failures): Closed
• Run HA dataset E2E test with new RestartPipelinedRegionStrategy: Resolved
• Add documentation for AdaptedRestartPipelinedRegionStrategyNG: Resolved
• Set jobmanager.execution.failover-strategy to region in default flink-conf.yaml: Closed