[FLINK-34143] Modify the effective strategy of `execution.batch.adaptive.auto-parallelism.default-source-parallelism`

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.19.0
• Fix Version/s: 1.19.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: xingbe
• Reporter: xingbe
• Created: 18/Jan/24 09:08
• Updated: 01/Feb/24 08:25
• Resolved: 24/Jan/24 02:31

Description

Currently, if users do not set the `execution.batch.adaptive.auto-parallelism.default-source-parallelism` configuration option, the AdaptiveBatchScheduler defaults to a parallelism of 1 for source vertices. In FLIP-379, the value of `execution.batch.adaptive.auto-parallelism.default-source-parallelism` will act as the upper bound for inferring dynamic source parallelism, and continuing with the current policy is no longer appropriate.

We plan to change the effectiveness strategy of `execution.batch.adaptive.auto-parallelism.default-source-parallelism`; when the user does not set this config option, we will use the value of `execution.batch.adaptive.auto-parallelism.max-parallelism` as the upper bound for source parallelism inference. If `execution.batch.adaptive.auto-parallelism.max-parallelism` is also not configured, the value of `parallelism.default` will be used as a fallback.

Issue Links

• links to: GitHub Pull Request #24170