[BEAM-3301] Go SplittableDoFn support

• Type: Improvement
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: sdk-go
• Assignee: Daniel Oliveira
• Reporter: Henning Rohde
• Created: 06/Dec/17 00:15
• Updated: 08/Dec/20 01:54
• Resolved: 08/Dec/20 01:54

Description

SDFs will be the only way to add streaming and liquid sharded IO for Go.

Design doc: https://s.apache.org/splittable-do-fn

Issue Links

• depends upon: BEAM-65 SplittableDoFn
• depends upon: BEAM-2939 Fn API SDF support
• links to: GitHub Pull Request #10778
• links to: GitHub Pull Request #10801
• links to: GitHub Pull Request #10991
• links to: GitHub Pull Request #11144
• links to: GitHub Pull Request #11179
• links to: GitHub Pull Request #11188
• links to: GitHub Pull Request #11225
• links to: GitHub Pull Request #11257
• links to: GitHub Pull Request #12628
• links to: GitHub Pull Request #12629

Sub-Tasks

• Add SDF validation code: Resolved
• Add SDF execution-time runners: Resolved
• Add user-facing Go SDF documentation.: Resolved
• Add capability to run SDFs on Flink Runner and Python FnApiRunner.: Resolved
• Add thread-safe restriction trackers.: Resolved
• Sharing restriction trackers between SDF ProcessElement executor and DataSource: Resolved
• Sub-element progress tracking and reporting in Go SDK.: Resolved
• Implement dynamic splitting support for SDFs in Go SDK.: Resolved
• Adjust Go SDF progress reporting to account for multi-window elements.: Resolved
• Fix multi-window SDF splitting in Go.: Resolved