Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-2889 Flink runs portable pipelines
  3. BEAM-3673

FlinkRunner: Harness manager for connecting operators to SDK Harnesses

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: runner-flink
    • Labels:
      None

      Description

      SDK harnesses require a common set of gRPC services to operate. The role of the harness manager is to provide a uniform interface that multiplexes data streams and auxiliary data between SDK environments and operators within a given job.

      Note that multiple operators may communicate with a single SDK environment to amortize container initialization cost. Environments are not shared between different jobs.

      The initial implementation will shell out to local docker, but the harness manager should eventually support working with externally-managed environments (e.g., created by Kubernetes).

        Attachments

          Activity

            People

            • Assignee:
              axelmagn Axel Magnuson
              Reporter:
              bsidhom Ben Sidhom
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: