[BEAM-1438] The default behavior for the Write transform doesn't work well with the Dataflow streaming runner

• Type: Bug
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.5.0
• Component/s: runner-dataflow
• Assignee: Unassigned
• Reporter: Reuven Lax
• Created: 08/Feb/17 21:00
• Updated: 13/Jan/22 17:02
• Resolved: 12/Jan/22 04:39

Description

If a Write specifies 0 output shards, that implies the runner should pick an appropriate sharding. The default behavior is to write one shard per input bundle. This works well with the Dataflow batch runner, but not with the streaming runner which produces large numbers of small bundles.

Issue Links

• relates to: BEAM-5865 Auto sharding of streaming sinks in FlinkRunner
• links to: GitHub Pull Request #1952
• links to: GitHub Pull Request #11850