Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.10.0
Description
In scheduler NG, scheduling strategies invokes SchedulerOperations#allocateSlotsAndDeploy(Collection<ExecutionVertexDeploymentOption>) to trigger scheduling of tasks.
However, EagerSchedulingStrategy and LazyFromSourcesSchedulingStrategy both invokes it by passing a batch of tasks, but requires the scheduling process to be conducted in 2 different ways:
- EagerSchedulingStrategy requires the batch of tasks to deploy after all of them have acquired slots. This is essential to avoid partition update RPCs in streaming job scheduling.
- LazyFromSourcesSchedulingStrategy requires tasks in the batch to allocate slot and get deployed individually, so that it can deploy a few tasks even if the slots is not enough for all tasks in the batch. This is helpful for batch job scheduling.
The scheduler then have to decide the scheduling pattern based whether the scheduling strategy is a LazyFromSourcesSchedulingStrategy. This is not good, as there can be more strategies in the future, and even customized scheduling strategies.
I think it's better to define the SchedulerOperations#allocateSlotsAndDeploy(Collection<ExecutionVertexDeploymentOption>) to be that all tasks in the batch need to be assigned and deployed together, like what we do for EagerSchedulingStrategy.
All scheduling strategies need to follow this rule. If tasks should be scheduled individually, the strategy should invoke allocateSlotsAndDeploy multiple times, one for each task. As a result, the LazyFromSourcesSchedulingStrategy needs to be adjusted for that.
Attachments
Issue Links
- blocks
-
FLINK-16430 FLIP-119 Pipelined Region Scheduling
-
- Closed
-
- links to
