Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1325

DataflowRunner support for Class-Path jars

Details

    • New Feature
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • runner-dataflow
    • None

    Description

      See https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/538

      The code that infers --filesToStage from the classpath does not seem to support a classpath jar. which uses a Class-Path manifest to indicate which other jars are needed at runtime. This is apparently a standard, approved way to pass the classpath.

      We need to add this support to DataflowRunner#detectClasspathResourcesToStage, but correctly representing the user's classpath continues to be challenging.

      (As FlinkRunner has copied this code, we should probably either pull the common utility somewhere public or update the FlinkRunner too?)

      Attachments

        Activity

          People

            Unassigned Unassigned
            dhalperi Dan Halperin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: