[BEAM-3247] Sample.any memory constraint

• Type: Improvement
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0
• Component/s: sdk-java-core
• Labels: Done
• Assignee: Neville Li
• Reporter: Neville Li
• Created: 24/Nov/17 21:10
• Updated: 06/Oct/20 21:59
• Resolved: 24/Sep/18 11:49

Description

Right now Sample.any converts the collection to an iterable view and take first n in a side input. This may require materializing the entire collection to disk and is potentially inefficient.
https://github.com/apache/beam/blob/v2.1.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Sample.java#L74

It can be fixed by applying a truncating `DoFn` first, then a combine into `List<T>` which limits the list size, and finally flattening the list.

Issue Links

• links to: GitHub Pull Request #4175