[BEAM-8608] Chain DoFns in Flink batch runner when possible.

• Type: Improvement
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.16.0
• Fix Version/s: 2.23.0
• Component/s: runner-flink
• Assignee: David Morávek
• Reporter: David Morávek
• Created: 11/Nov/19 15:03
• Updated: 21/Jul/20 23:24
• Resolved: 01/Jun/20 07:47

Description

Right now, in Batch runner, DoFn is executed using MapPartition operator (FlinkDoFnFunction), which doesn't have chained driver implementation.

We need to reimplement DoFnFunction with FlatMap to allow chaining.

Attached is the execution graph for the same pipeline, before and after the patch.

Issue Links

• depends upon: FLINK-14709 Allow outputting elements in close method of chained drivers.
• is depended upon by: BEAM-8848 Flink: Memory efficient GBK implementation for batch runner
• links to: GitHub Pull Request #11858