[BEAM-1325] DataflowRunner support for Class-Path jars

• Type: New Feature
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-dataflow
• Assignee: Unassigned
• Reporter: Dan Halperin
• Created: 26/Jan/17 14:41
• Updated: 03/Jun/22 16:24

Description

See https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/538

The code that infers --filesToStage from the classpath does not seem to support a classpath jar. which uses a Class-Path manifest to indicate which other jars are needed at runtime. This is apparently a standard, approved way to pass the classpath.

We need to add this support to DataflowRunner#detectClasspathResourcesToStage, but correctly representing the user's classpath continues to be challenging.

(As FlinkRunner has copied this code, we should probably either pull the common utility somewhere public or update the FlinkRunner too?)