[BEAM-14171] CoGroupByKey loses values with large groups on Dataflow v1

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Affects Version/s: 2.36.0, 2.37.0
• Fix Version/s: 2.38.0
• Component/s: runner-dataflow, sdk-java-core

Language

Java

• Assignee: Robert Bradshaw
• Reporter: Niel Markwick
• Created: 24/Mar/22 19:48
• Updated: 31/Mar/22 07:22
• Resolved: 31/Mar/22 07:22

Description

CoGroupByKey can lose elements - replacing them with null values when a group is large (>10,000 elements).

 

This only occurs in dataflow v1, not dataflow-v2 runner

Possibly related to BEAM-13541.

 

https://lists.apache.org/thread/5y56kbgm3q0m1byzf7186rrkomrcfldm

 

 

 

Issue Links

• links to: GitHub Pull Request #17180
• links to: GitHub Pull Request #17198