[FLINK-11299] Decide the result partition type dynamically

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Abandoned
• Component/s: Runtime / Coordination
• Assignee: shuai.xu
• Reporter: shuai.xu
• Created: 10/Jan/19 07:19
• Updated: 31/Jul/20 10:22
• Resolved: 31/Jul/20 10:22

Description

Now, the result partition type is decided when compiling JobGraph at the client. This's to say, whether to output the result of a task to its consumers by PIPELINED or BLOCKING is decided at client and can not be changed any more. However, it is usually seen in batch jobs that a task is configured PIPELINED but its consumers can not be started due to lack of resource, this will lead to failover now. So we proposal to support deciding the result partition type dynamically during runtime according to the resource of cluster and so on to increase the success rate of batch job.

Issue Links

• is related to: FLINK-10433 Stepwise creation of the ExecutionGraph sub-structures