[BEAM-4565] Hot key fanout should not distribute keys to all shards.

• Type: Task
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Affects Version/s: 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0
• Component/s: sdk-java-core, sdk-py-core
• Labels: Clarified
• Assignee: Unassigned
• Reporter: Robert Bradshaw
• Created: 14/Jun/18 22:44
• Updated: 03/Jun/22 22:10

Description

The goal is to reduce the number of value sent to a single post-GBK worker. If combiner lifting happens, each bundle will sends a single value per sub-key, causing an N-fold blowup in shuffle data and N reducers with the same amount of data to consume as the single reducer in the non-fanout case. 

Issue Links

• links to: GitHub Pull Request #5649