[FLINK-4256] Fine-grained recovery

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.9.0
• Component/s: Runtime / Coordination
• Assignee: Stephan Ewen
• Reporter: Stephan Ewen
• Created: 22/Jul/16 16:37
• Updated: 02/Oct/19 17:45
• Resolved: 16/Aug/19 10:41

Description

When a task fails during execution, Flink currently resets the entire execution graph and triggers complete re-execution from the last completed checkpoint. This is more expensive than just re-executing the failed tasks.

In many cases, more fine-grained recovery is possible.

The full description and design is in the corresponding FLIP.

https://cwiki.apache.org/confluence/display/FLINK/FLIP-1+%3A+Fine+Grained+Recovery+from+Task+Failures

The detail desgin for version1 is https://docs.google.com/document/d/1_PqPLA1TJgjlqz8fqnVE3YSisYBDdFsrRX_URgRSj74/edit#

Issue Links

• is blocked by: FLINK-4322 Unify CheckpointCoordinator and SavepointCoordinator
• relates to: FLINK-13371 Release partitions in JM if producer restarts
• relates to: FLINK-12069 Add proper lifecycle management for intermediate result partitions
• relates to: FLINK-10288 Failover Strategy improvement
• links to: GitHub Pull Request #3539