[BEAM-4147] Abstractions for artifact delivery via arbitrary storage backends

• Type: New Feature
• Status: Resolved
• Priority: P2
• Resolution: Resolved
• Fix Version/s: 2.6.0
• Component/s: runner-core
• Assignee: Axel Magnuson
• Reporter: Ben Sidhom
• Created: 19/Apr/18 22:23
• Updated: 16/May/20 13:41
• Resolved: 01/Aug/18 22:30

Description

We need a way to wire in arbitrary runner artifact storage backends into the job server and through to artifact staging on workers. This requires some new abstractions in front of the job service.

Issue Links

• is depended upon by: BEAM-2889 Flink runs portable pipelines

Sub-Tasks

• ArtifactStagingService that stages to a distributed filesystem: Resolved
• ArtifactRetrievalService that retrieves artifacts from a distributed filesystem: Resolved