[BEAM-3089] Issue with setting the parallelism at client level using Flink runner

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.8.0
• Component/s: runner-flink
• Labels: docker flink parallel-deployment
• Environment: I am using Flink 1.2.1 running on Docker, with Task Managers distributed across different VMs as part of a Docker Swarm.
• Flags: Patch, Important
• Assignee: Maximilian Michels
• Reporter: Thalita Vergilio
• Created: 22/Oct/17 17:51
• Updated: 16/May/20 14:00
• Resolved: 20/Sep/18 13:17

Description

When uploading an Apache Beam application using the Flink Web UI, the parallelism set at job submission doesn't get picked up. The same happens when submitting a job using the Flink CLI.

In both cases, the parallelism ends up defaulting to 1.

When I set the parallelism programmatically within the Apache Beam code, it works: flinkPipelineOptions.setParallelism(4);

I suspect the root of the problem may be in the org.apache.beam.runners.flink.DefaultParallelismFactory class, as it checks for Flink's GlobalConfiguration, which may not pick up runtime values passed to Flink, then defaults to 1 if it doesn't find anything.

Any ideas on how this could be fixed or worked around? I need to be able to change the parallelism dynamically, so the programmatic approach won't really work for me, nor will setting the Flink configuration at system level.

Issue Links

• is cloned by: BEAM-7186 LTS backport: Issue with setting the parallelism at client level using Flink runner
• links to: GitHub Pull Request #4766
• links to: GitHub Pull Request #6426