[FLINK-32027] Batch jobs could hang at shuffle phase when max parallelism is really large

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.16.0, 1.17.0, 1.16.1
• Fix Version/s: 1.16.2, 1.18.0, 1.17.1
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Weijie Guo
• Reporter: Yun Tang
• Created: 08/May/23 03:17
• Updated: 10/May/23 03:00
• Resolved: 10/May/23 03:00

Description

In batch stream mode with adaptive batch schedule mode, If we set the max parallelism large as 32768 (pipeline.max-parallelism), the job could hang at the shuffle phase:

It would hang for a long time and show "No bytes sent":

After some time to debug, we can see the downstream operator did not receive the end-of-partition event.

Issue Links

• is caused by: FLINK-28519 Fix the bug that SortMergeResultPartitionReadScheduler may not read data sequentially
• links to: GitHub Pull Request #22549