[BEAM-10164] Flink: Memory efficient combine implementation for batch runner

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.22.0
• Component/s: runner-flink
• Assignee: David Morávek
• Reporter: David Morávek
• Created: 01/Jun/20 11:49
• Updated: 24/Jul/20 19:52
• Resolved: 03/Jun/20 13:33

Description

Current Combine implementation assumes that all input values for a single key (on both map and reduce side) fit in memory as it needs to sort them by window before combining.

We can easily optimize this for non-merging windows by pre-grouping elements by (K, W) tuples.

Issue Links

• is related to: BEAM-8848 Flink: Memory efficient GBK implementation for batch runner
• links to: GitHub Pull Request #11889