[BEAM-11317] SDF BoundedSource consumer is ont able to split data from the runner

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: io-py-common
• Assignee: Unassigned
• Reporter: Pablo Estrada
• Created: 21/Nov/20 05:58
• Updated: 04/Jun/22 18:18

Description

This is affecting BigQuery jobs that export into a small number of files and have a large overhead per element in the same stage.

This makes stages slow to execute and unable to split work - so jobs end up being very long. Should look into this.