Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-802

Support Dynamic PipelineOptions for python

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: sdk-py-core
    • Labels:
      None

      Description

      Goal: Enable users to run pipelines from templates filled via CL (pipeline options)
      Background: Currently, the Runner creates the JSON pipeline description which can be sent to the worker as is, since everything is already defined there (with links to gs:// for input and binaries). With the parametrized approach, those descriptions are empty and filled by the user or defaulted, so the pipeline needs to be stored somewhere first until the values become available.
      Tasks:
      1- Create template-style pipeline description (TemplateRunner)
      The graph description is now a template (some parts are not filled) that needs to be saved.
      2- Define values to inject to the template (ValueProviders API)
      The placeholders can be filled with default values (static) or with dynamic key/value pairs provided at runtime (dynamic)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mariagh María GH
                Reporter:
                mariagh María GH
              • Votes:
                2 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - 1,680h
                  1,680h
                  Remaining:
                  Remaining Estimate - 1,680h
                  1,680h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified