[BEAM-12865] Allow customising batch duration when streaming with WriteToBigQuery

• Type: New Feature
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Affects Version/s: Not applicable
• Fix Version/s: 2.36.0
• Component/s: io-py-gcp
• Assignee: Unassigned
• Reporter: Quentin Sommer
• Created: 09/Sep/21 14:55
• Updated: 13/Apr/23 11:08
• Resolved: 04/Jan/22 20:11

Description

Hi,

We allow customising the batch_size when streaming to BigQuery but the batch duration (used by GroupIntoBatches) is set to DEFAULT_BATCH_BUFFERING_DURATION_LIMIT_SEC (0.2)

I'd like to add the option to specify the batch duration to allow better batching for scenarios with little data throughput.
It will use the triggering_frequency param already used when doing batch file loads

Issue Links

• links to: GitHub Pull Request #15489
• links to: GitHub Pull Request #16315