[BEAM-3883] Python SDK stages artifacts when talking to job server

• Type: Sub-task
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.6.0
• Component/s: sdk-py-core
• Assignee: Ankur Goenka
• Reporter: Ben Sidhom
• Created: 19/Mar/18 22:46
• Updated: 24/Jul/20 20:11
• Resolved: 28/Jun/18 23:09

Description

The Python SDK does not currently stage its user-defined functions or dependencies when talking to the job API. Artifacts that need to be staged include the user code itself, any SDK components not included in the container image, and the list of Python packages that must be installed at runtime.

 

Artifacts that are currently expected can be found in the harness boot code: https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.

Issue Links

• is depended upon by: BEAM-4204 Python: PortableRunner - p.run() via given JobService
• is depended upon by: BEAM-2889 Flink runs portable pipelines
• links to: GitHub Pull Request #5251
• links to: GitHub Pull Request #5273
• links to: GitHub Pull Request #5817