[FLINK-30631] Limit the max number of subpartitons consumed by each downstream task

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.17.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Lijie Wang
• Reporter: Lijie Wang
• Created: 11/Jan/23 08:46
• Updated: 16/Jan/23 09:17
• Resolved: 16/Jan/23 09:17

Description

In the current implementation(FLINK-25035), when the upstream vertex parallelism is much greater than the downstream vertex parallelism, it may lead to a large number of channels in the downstream tasks(for example, A -> B, all to all edge, max parallelism is 1000. If parallelism of A is 1000, parallelism of B is decided to be 1, then the only subtask of B will consume 1000 * 1000 subpartitions), resulting in a large overhead for processing channels.

In this ticket, we temporarily address this issue by limiting the max number of subpartitons consumed by each downstream task. The ultimate solution should be to support single channel consume multiple subpartitons.

Issue Links

• links to: GitHub Pull Request #21646