[FLINK-14236] Make LazyFromSourcesSchedulingStrategy do lazy scheduling based on partition state only

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Won't Do
• Affects Version/s: 1.10.0
• Component/s: Runtime / Coordination
• Assignee: Unassigned
• Reporter: Zhu Zhu
• Created: 26/Sep/19 14:51
• Updated: 28/Aug/20 12:03
• Resolved: 28/Aug/20 12:03

Description

The philosophy of lazy_from_sources scheduling is to schedule a task when its inputs are ready, i.e. when needed result partitions are consumable. It actually does not directly care whether a task is finished, though a finished task will lead its partitions to be consumable.

The LazyFromSourcesSchedulingStrategy currently does lazy scheduling in two ways:
1. trigger scheduling on FINISHED execution state for blocking partition consumers
2. trigger scheduling on partition consumable events for pipelined partition consumers
This makes the scheduling decision a bit messy. And it also requires the LazyFromSourcesSchedulingStrategy to manage the partition state by itself, to decide when a partition is consumable, which is not good.

I'd propose to make LazyFromSourcesSchedulingStrategy do lazy scheduling only based on partition state. This can make the scheduling decision making clearer and relieve LazyFromSourcesSchedulingStrategy from partition state management.

This work requires FLINK-14234 so that all partition consumable events can be notified.

Issue Links

• is blocked by: FLINK-14234 All partition consumable events should be notified to SchedulingStrategy (SchedulerNG)