[BEAM-1631] Flink runner: submit job to a Flink-on-YARN cluster

• Type: New Feature
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-flink
• Assignee: Unassigned
• Reporter: Davor Bonaci
• Created: 06/Mar/17 21:46
• Updated: 03/Jun/22 16:33

Description

As far as I understand, running Beam pipelines on a Flink cluster can be done in two ways:

• Run directly with a Flink runner, and specifying --flinkMaster pipeline option via, say, mvn exec.
• Produce a bundled JAR, and use bin/flink to submit the same pipeline.

These two ways are equivalent, and work well on a standalone Flink cluster.

Submitting to a Flink-on-YARN is more complicated. You can still produce a bundled JAR, and use bin/flink -yid <applicationid> to submit such a job. However, that seems impossible with a Flink runner directly.

If so, we should add the ability to the Flink runner to submit a job to a Flink-on-YARN cluster directly.