Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7186

LTS backport: Issue with setting the parallelism at client level using Flink runner

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • 2.0.0
    • 2.7.1
    • runner-flink
    • I am using Flink 1.2.1 running on Docker, with Task Managers distributed across different VMs as part of a Docker Swarm.
    • Patch, Important

    Description

      When uploading an Apache Beam application using the Flink Web UI, the parallelism set at job submission doesn't get picked up. The same happens when submitting a job using the Flink CLI.

      In both cases, the parallelism ends up defaulting to 1.

      When I set the parallelism programmatically within the Apache Beam code, it works: flinkPipelineOptions.setParallelism(4);

      I suspect the root of the problem may be in the org.apache.beam.runners.flink.DefaultParallelismFactory class, as it checks for Flink's GlobalConfiguration, which may not pick up runtime values passed to Flink, then defaults to 1 if it doesn't find anything.

      Any ideas on how this could be fixed or worked around? I need to be able to change the parallelism dynamically, so the programmatic approach won't really work for me, nor will setting the Flink configuration at system level.

      Attachments

        Issue Links

          Activity

            People

              mxm Maximilian Michels
              mxm Maximilian Michels
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: