[BEAM-65] SplittableDoFn

• Type: New Feature
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: beam-model
• Assignee: Eugene Kirpichov
• Reporter: Dan Halperin
• Created: 25/Feb/16 16:26
• Updated: 24/Jul/20 20:53
• Resolved: 13/Feb/18 19:55

Description

SplittableDoFn is a proposed enhancement for "dynamically splittable work" to the Beam model.

Among other things, it would allow a unified implementation of bounded/unbounded sources with dynamic work rebalancing and the ability to express multiple scalable steps (e.g., global expansion -> file sizing & parsing -> splitting files into independently-processable blocks) via composition rather than inheritance.

This would make it much easier to implement many types of sources, to modify and reuse existing sources. Also, it would improve scalability of the Beam model by moving things like splitting a source from the control plane (where it is today – glob -> List<FileBasedSource> sent over service APIs) into the data plane (PCollection<Glob> -> PCollection<FileName> -> ...).

Issue Links

• incorporates: BEAM-1903 Splittable DoFn should report watermarks via ProcessContext
• incorporates: BEAM-1904 Remove DoFn.ProcessContinuation
• incorporates: BEAM-2301 Standard expansion of SDF should be in runners-core-construction
• incorporates: BEAM-1377 Support Splittable DoFn in Dataflow streaming runner
• incorporates: BEAM-1855 Support Splittable DoFn in Flink Streaming runner
• is depended upon by: BEAM-3301 Go SplittableDoFn support
• is related to: BEAM-644 Primitive to shift the watermark while assigning timestamps
• links to: GitHub Pull Request #896
• links to: GitHub Pull Request #1824
• links to: GitHub Pull Request #1895
• links to: GitHub Pull Request #2462