[FLINK-28519] Fix the bug that SortMergeResultPartitionReadScheduler may not read data sequentially

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.16.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Yuxin Tan
• Reporter: Yingjie Cao
• Created: 12/Jul/22 12:15
• Updated: 08/May/23 11:54
• Resolved: 29/Jul/22 11:59

Description

Currently, the SortMergeResultPartitionReadScheduler always gets all active subpartition readers and read at most one data region for them. It is common that some subpartitions are requested before others and their region indexes are ahead of others. If all region data of a subpartition can be read in one round, some subpartition readers will always ahead of others which will cause random IO. This patch fixes this case by polling one subpartition reader at a time.

Issue Links

• causes: FLINK-32027 Batch jobs could hang at shuffle phase when max parallelism is really large
• links to: GitHub Pull Request #20320