[FLINK-33534] PipelineOptions.PARALLELISM_OVERRIDES is not picked up from jar submission request

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.18.0, 1.17.1
• Fix Version/s: 1.19.0
• Component/s: Runtime / Configuration, Runtime / REST
• Labels: pull-request-available
• Assignee: Yunfeng Zhou
• Reporter: Gyula Fora
• Created: 13/Nov/23 05:38
• Updated: 20/Dec/23 03:55
• Resolved: 19/Dec/23 01:24

Description

PARALLELISM_OVERRIDES are currently only applied when they are part of the JobManager / Cluster configuration.

When this config is provided as part of the JarRunRequestBody it is completely ignored and does not take effect. 

The main reason is that the dispatcher reads this value from it's own configuration object and does not include the extra configs passed through the rest request.

This is a blocker for supporting the autoscaler properly for FlinkSessionJobs in the autoscaler

Issue Links

• links to: GitHub Pull Request #23944
• links to: GitHub Pull Request #23953