Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-551

Support Dynamic PipelineOptions

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Not applicable
    • Component/s: sdk-java-core
    • Labels:
      None

      Description

      During the graph construction phase, the given SDK generates an initial
      execution graph for the program. At execution time, this graph is
      executed, either locally or by a service. Currently, Beam only supports
      parameterization at graph construction time. Both Flink and Spark supply
      functionality that allows a pre-compiled job to be run without SDK
      interaction with updated runtime parameters.

      In its current incarnation, Dataflow can read values of PipelineOptions at
      job submission time, but this requires the presence of an SDK to properly
      encode these values into the job. We would like to build a common layer
      into the Beam model so that these dynamic options can be properly provided
      to jobs.

      Please see
      https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
      for the high-level model, and
      https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
      for
      the specific API proposal.

        Attachments

          Activity

            People

            • Assignee:
              sgmc@google.com Sam McVeety
              Reporter:
              sam.mcveety@gmail.com Sam McVeety
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: