[BEAM-11303] DataFrame GroupBy().size() aggregation produces incorrect results

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.25.0
• Fix Version/s: 2.26.0
• Component/s: sdk-py-core
• Assignee: Brian Hulette
• Reporter: Brian Hulette
• Created: 18/Nov/20 21:07
• Updated: 15/May/21 17:59
• Resolved: 15/May/21 17:59

Description

size is treated as a liftable aggregation which assumes it is commutative and associative, but it's not actually associative. It can be lifted, but the post agg step needs to be a sum.
This means the size aggregation will produce incorrect results.

Issue Links

• links to: GitHub Pull Request #13379
• links to: GitHub Pull Request #13386