[FLINK-10573] Support task revocation

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Runtime / Coordination
• Assignee: ryantaocer
• Reporter: JIN SUN
• Created: 16/Oct/18 22:08
• Updated: 03/Jul/20 09:23
• Resolved: 03/Jul/20 09:23

Description

In Batch Mode, When a downstream task has a partition missing failure, which indicate the output of upstream task has been lost. To make the job success we need to rerun the upstream task to reproduce the data, which we call task revocation (revoke the success of upstream task)

For revocation, we need to identify the partition missing issue, and it is better to detect the missing partition accurately:

• Ideally, it makes things much easier if we get a specific exception indicating that the data source is missing
• When a task got an IOException, it doesn’t mean the source data has issues. It might also be related to target task, such as that the target task has network issues.
• If multiple tasks cannot read the same source, it is highly likely the source data is missing.

Issue Links

• duplicates: FLINK-6227 Introduce the abstract PartitionException for downstream task failure