[FLINK-32288] Improve the scheduling performance of AdaptiveBatchScheduler

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.18.0
• Fix Version/s: 1.18.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: xingbe
• Reporter: xingbe
• Created: 08/Jun/23 09:05
• Updated: 20/Jun/23 03:12
• Resolved: 20/Jun/23 03:12

Description

After adding the benchmark of AdaptiveBatchScheduler in FLINK-30480, we noticed a regression in the performance of SchedulingDownstreamTasksInBatchJobBenchmark#SchedulingDownstreamTasks. When scheduling a batch job with a parallelism of 4000*4000, the time spent increased from 32ms to 1336ms on my local PC.

To improve the performance, we can optimize the traversal by checking if the consumedPartitionGroups have finished all its partitions.

Issue Links

• is duplicated by: FLINK-32306 Multiple batch scheduler performance regressions
• is related to: FLINK-30480 Add benchmarks for adaptive batch scheduler
• links to: GitHub Pull Request #22798