[FLINK-31996] Chaining operators with different max parallelism prevents rescaling

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.18.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: David Morávek
• Reporter: David Morávek
• Created: 04/May/23 08:22
• Updated: 16/Oct/23 11:26
• Resolved: 06/Jul/23 13:43

Description

We might chain operators with different max parallelism together if they are set to have the same parallelism initially.

When we decide to rescale the JobGraph vertices (using AdaptiveScheduler), we're gapped by the lowest maxParallelism of the operator chain. This is especially visible with things like CollectSink, TwoPhaseCommitSink, CDC, and a GlobalCommiter with maxParallelism set to 1.

 

An obvious solution would be to prevent the chaining of operators with different maxParallelism, but we need to double-check this doesn't introduce a breaking change.

Issue Links

• is duplicated by: FLINK-32022 Source level scaling is not applied to operators with chaining applied
• is related to: FLINK-31316 FLIP-291: Externalized Declarative Resource Management
• links to: GitHub Pull Request #22520