[BEAM-6611] A Python Sink for BigQuery with File Loads in Streaming

• Type: Improvement
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.16.0
• Component/s: sdk-py-core
• Labels: gsoc gsoc2019 mentor
• Assignee: Tanay Tummalapalli
• Reporter: Pablo Estrada
• Created: 06/Feb/19 17:49
• Updated: 13/Apr/23 10:56
• Resolved: 04/Sep/19 17:37

Description

The Java SDK supports a bunch of methods for writing data into BigQuery, while the Python SDK supports the following:

• Streaming inserts for streaming pipelines As seen in [bigquery.py and BigQueryWriteFn
• File loads for batch pipelines As implemented in [PR 7655

Qucik and dirty early design doc: https://s.apache.org/beam-bqfl-py-streaming
The Java SDK also supports File Loads for Streaming pipelines see BatchLoads application.

File loads have the advantage of being much cheaper than streaming inserts (although they also are slower for the records to show up in the table).

Issue Links

• links to: GitHub Pull Request #8871
• links to: GitHub Pull Request #9216