[FLINK-29501] Allow overriding JobVertex parallelisms during job submission

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.17.0
• Component/s: Runtime / Configuration, Runtime / REST
• Labels: pull-request-available
• Assignee: Maximilian Michels
• Reporter: Maximilian Michels
• Created: 04/Oct/22 14:13
• Updated: 23/Jan/23 16:02
• Resolved: 23/Nov/22 15:58

Description

It is a common scenario that users want to make changes to the parallelisms in the JobGraph. For example, because they discover that the job needs more or less resources. There is the option to do this globally via the job parallelism. However, for fine-tuned jobs jobs with potentially many branches, tuning on the job vertex level is required.

This is to propose a way such that users can apply a mapping jobVertexId => parallelism before the job is submitted without having to modify the JobGraph manually.

One way to achieving this would be to add an optional map field to the Rest API jobs endpoint. However, in deployment modes like the application mode, this might not make sense because users do not have control the rest endpoint.

Similarly to how other job parameters are passed in the application mode, we propose to add the overrides as a configuration parameter.

Issue Links

• causes: FLINK-30213 The edge is wrong when the vertex parallelism is changed
• relates to: FLINK-30773 Add API for rescaling of jobs based on per-vertex parallelism overrides
• links to: GitHub Pull Request #21023