[FLINK-14234] All partition consumable events should be notified to SchedulingStrategy (SchedulerNG)

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Won't Do
• Affects Version/s: 1.10.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Zhu Zhu
• Reporter: Zhu Zhu
• Created: 26/Sep/19 13:01
• Updated: 28/Aug/20 12:04
• Resolved: 28/Aug/20 12:04

Description

SchedulingStrategy requires partition consumable notification to make scheduling decisions.
According to SchedulingStrategy#onPartitionConsumable definition, all partition consumable events should be notified to SchedulingStrategy, including those from TMs (pipelined partitions consumable for data produced) and from within JM(blocking partitions consumable for producer finished).

In this way, the LazyFromSourcesSchedulingStrategy does not need to maintain the result partition status by itself. InputDependencyConstraintChecker can be simplified a lot in this way.
Besides that, LazyFromSourcesSchedulingStrategy does not need to be aware of result partition types(PIPELINED/BLOCKING) in this way.

It would also simplify the input checking for pipelined region scheduling.

More details see here.

Issue Links

• blocks: FLINK-14236 Make LazyFromSourcesSchedulingStrategy do lazy scheduling based on partition state only
• links to: GitHub Pull Request #11691