[BEAM-68] Support for limiting parallelism of a step

• Type: New Feature
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: beam-model
• Assignee: Unassigned
• Reporter: Dan Halperin
• Created: 25/Feb/16 16:50
• Updated: 03/Jun/22 16:47

Description

Users may want to limit the parallelism of a step. Two classic uses cases are:

• User wants to produce at most k files, so sets TextIO.Write.withNumShards(k).
• External API only supports k QPS, so user sets a limit of k/(expected QPS/step) on the ParDo that makes the API call.

Unfortunately, there is no way to do this effectively within the Beam model. A GroupByKey with exactly k keys will guarantee that only k elements are produced, but runners are free to break fusion in ways that each element may be processed in parallel later.

To implement this functionaltiy, I believe we need to add this support to the Beam Model.

Issue Links

• is duplicated by: BEAM-159 Support fixed number of shards in sinks