[BEAM-4786] Distinct has bad parallelism characteristics

• Type: Bug
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: sdk-java-core
• Labels: Clarified ccoss2019
• Assignee: Unassigned
• Reporter: Pablo Estrada
• Created: 13/Jul/18 21:21
• Updated: 03/Jun/22 22:08

Description

Distinct groups first and then drops extra elements. Should drop elements in the mappers, and later in reducers.

https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Distinct.java#L100