[FLINK-19693] Scheduler Change for Approximate Local Recovery to Restart Downstream of a Failed Task

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.12.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Yuan Mei
• Reporter: Yuan Mei
• Created: 18/Oct/20 12:01
• Updated: 09/Nov/20 16:57
• Resolved: 08/Nov/20 17:45

Description

Enables downstream failover for approximate local recovery.

That says if a task fails, all its downstream tasks restart, including itself. This is achieved by reusing the existing RestartPipelinedRegionFailoverStrategy --- treat each individual task connected by ResultPartition.Pipelined_Approximate as a separate region.

 

It introduces an attribute "reconnectable" in ResultPartitionType to indicate whether the partition is reconnectable. Notice that this is only a temporary solution for now. It will be removed after:

1. Approximate local recovery has its won failover strategy to restart the failed set of tasks instead of restarting downstream of failed tasks depending on {[@link|https://github.com/code] RestartPipelinedRegionFailoverStrategy}
2. FLINK-19895: Unify the life cycle of ResultPartitionType Pipelined Family. There is also a good discussion on this in FLINK-19632.

Issue Links

• links to: GitHub Pull Request #13880