Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8660

Override returned artifact staging endpoint

    XMLWordPrintableJSON

    Details

      Description

      When running Beam Python pipelines on Flink/Spark/etc, we connect the SDK to the job server using the job_endpoint option. The job server then returns the address of the artifact staging endpoint to the SDK.

      This is problematic when running the job server in network environments where the job server is not aware of its external hostname, for example Kubernetes. In this case, the job server will return something like localhost:8098, which might not be correct. While we do have a --job-host option, this is used both internally and externally, and the internal and external host names may not be the same.

      One solution would be to configure two separate host names in the job server. However I do not prefer this option because of the complexity it adds.

      The more straightforward solution is to add an option to Python that overrides the artifact staging endpoint returned by the server.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ibzib Kyle Weaver
                Reporter:
                ibzib Kyle Weaver
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m