[BEAM-2884] Dataflow runs portable pipelines

• Type: Improvement
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.8.0
• Component/s: runner-dataflow
• Labels: portability
• Assignee: Unassigned
• Reporter: Henning Rohde
• Created: 11/Sep/17 17:22
• Updated: 16/May/20 13:50
• Resolved: 21/Sep/18 21:17

Description

Dataflow should run pipelines using the full portability API as currently defined:

https://s.apache.org/beam-fn-api
https://s.apache.org/beam-runner-api
https://s.apache.org/beam-job-api
https://s.apache.org/beam-fn-api-container-contract

This issue tracks its adoption of the portability framework. New Fn API and other features will be tracked separately.

Issue Links

• depends upon: BEAM-2876 Add provision api proto
• depends upon: BEAM-2881 GCS artifact server proxy
• is depended upon by: BEAM-2896 Portability milestone: wordcount runs everywhere
• links to: GitHub Pull Request #3858
• links to: GitHub Pull Request #3859
• links to: GitHub Pull Request #3887
• links to: GitHub Pull Request #4089

Sub-Tasks

• Use portable ParDoPayload in DataflowRunner: Resolved
• Propagate pipeline protos through Dataflow API from Python: Resolved
• Propagate pipeline protos through Dataflow API from Go: Resolved
• Use portable CombinePayload in Java DataflowRunner: Resolved
• Use portable ParDoPayload for SDF in DataflowRunner: Resolved