[BEAM-2880] Artifact server proxies

• Type: Improvement
• Status: Resolved
• Priority: P3
• Resolution: Won't Fix
• Fix Version/s: Not applicable
• Component/s: beam-model
• Labels: portability
• Assignee: Henning Rohde
• Reporter: Henning Rohde
• Created: 11/Sep/17 16:53
• Updated: 16/May/20 13:20
• Resolved: 22/Sep/18 15:44

Description

As per https://s.apache.org/beam-fn-api-container-contract, we should add runner-agnostic (dockerized) artifact proxies for various runner environments. These proxies implement the server side of the artifact API.

We'll likely need proxies for GCS, S3, Azure Storage, and local. If the proxies are implemented in a statically-linked language like Go, say, then they do not have a dependency footprint and can easily be used as native binaries or in dockerized form – at the discretion of the runner.

Issue Links

• is depended upon by: BEAM-2889 Flink runs portable pipelines
• links to: GitHub Pull Request #6466

Sub-Tasks

• GCS artifact server proxy: Resolved
• S3 artifact server proxy: Resolved