[BEAM-4228] The FlinkRunner shouldn't require all of the values for a key to fit in memory

• Type: New Feature
• Status: Triage Needed
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-flink
• Assignee: Unassigned
• Reporter: Thomas Groh
• Created: 02/May/18 22:04
• Updated: 22/Nov/24 07:50

Description

The use of a reducer that adds all of the elements that it consumes to a list is the primary way in which this occurs - if instead, we produce a filtered iterable, or a collection of filtered iterables, we can lazily iterate over all of the contained elements without having to buffer all of the elements.

 

For an example of where this occurs, see Concatenate in  FlinkBatchPortablePipelineTranslator.

Issue Links

• is related to: BEAM-8848 Flink: Memory efficient GBK implementation for batch runner