[BEAM-8312] Flink portable pipeline jars do not need to stage artifacts remotely

• Type: Improvement
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.17.0
• Component/s: runner-flink
• Labels: portability-flink
• Assignee: Kyle Weaver
• Reporter: Kyle Weaver
• Created: 25/Sep/19 00:10
• Updated: 13/Apr/23 10:57
• Resolved: 09/Oct/19 17:52

Description

Currently, Flink job jars re-stage all artifacts at runtime (on the JobManager) by using the usual BeamFileSystemArtifactRetrievalService [1]. However, since the manifest and all the artifacts live on the classpath of the jar, and everything from the classpath is copied to the Flink workers anyway [2], it should not be necessary to do additional artifact staging. We could replace BeamFileSystemArtifactRetrievalService in this case with a simple ArtifactRetrievalService that just pulls the artifacts from the classpath.

 

 [1] https://github.com/apache/beam/blob/340c3202b1e5824b959f5f9f626e4c7c7842a3cb/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java

[2] https://github.com/apache/beam/blob/2f1b56ccc506054e40afe4793a8b556e872e1865/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java#L93

Issue Links

• links to: GitHub Pull Request #9681
• links to: GitHub Pull Request #9717