Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1325

DataflowRunner support for Class-Path jars

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: runner-dataflow
    • Labels:
      None

      Description

      See https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/538

      The code that infers --filesToStage from the classpath does not seem to support a classpath jar. which uses a Class-Path manifest to indicate which other jars are needed at runtime. This is apparently a standard, approved way to pass the classpath.

      We need to add this support to DataflowRunner#detectClasspathResourcesToStage, but correctly representing the user's classpath continues to be challenging.

      (As FlinkRunner has copied this code, we should probably either pull the common utility somewhere public or update the FlinkRunner too?)

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              dhalperi Dan Halperin
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: