[FLINK-33768] FLIP-379: Support dynamic source parallelism inference for batch jobs

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.19.0
• Fix Version/s: 1.19.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available

Release Note

In Flink 1.19, we have supported dynamic source parallelism inference for batch jobs, which allows source connectors to dynamically infer the parallelism based on the actual amount of data to consume. This feature is a significant improvement over previous versions, which only assigned a fixed default parallelism to source vertices.

Source connectors need to implement the inference interface to enable dynamic parallelism inference. Currently, the FileSource connector has already been developed with this functionality in place.

Additionally, the configuration `execution.batch.adaptive.auto-parallelism.default-source-parallelism` will be used as the upper bound of source parallelism inference. And now it will not default to 1. Instead, if it is not set, the upper bound of allowed parallelism set via `execution.batch.adaptive.auto-parallelism.max-parallelism` will be used. If that configuration is also not set, the default parallelism set via `parallelism.default` or StreamExecutionEnvironment#setParallelism() will be used instead.

• Assignee: xingbe
• Reporter: xingbe
• Created: 07/Dec/23 01:58
• Updated: 07/Feb/24 10:02
• Resolved: 01/Feb/24 09:31

Description

Currently, for JobVertices without parallelism configured, the AdaptiveBatchScheduler dynamically infers the vertex parallelism based on the volume of input data. Specifically, for Source vertices, it uses the value of `execution.batch.adaptive.auto-parallelism.default-source-parallelism` as the fixed parallelism. If this is not set by the user, the default value of 1  is used as the source parallelism, which is actually a temporary implementation solution.

We aim to support dynamic source parallelism inference for batch jobs. More details see FLIP-379.

Issue Links

• is related to: FLINK-34356 Release Testing: Verify FLINK-33768 Support dynamic source parallelism inference for batch jobs
• links to: GitHub Pull Request #24087
• mentioned in: Page Loading...

Sub-Tasks

• Update the documentation and configuration description about dynamic source parallelism inference: Closed
• Modify the effective strategy of `execution.batch.adaptive.auto-parallelism.default-source-parallelism`: Closed
• File source connector support dynamic source parallelism inference in batch jobs: Closed