[BEAM-7742] BigQuery File Loads to work well with load job size limits

• Type: Improvement
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.16.0
• Component/s: io-py-gcp
• Assignee: Tanay Tummalapalli
• Reporter: Pablo Estrada
• Created: 15/Jul/19 16:55
• Updated: 13/Apr/23 10:56
• Resolved: 04/Sep/19 19:59

Description

Load jobs into BigQuery have a number of limitations: https://cloud.google.com/bigquery/quotas#load_jobs

 

Currently, the python BQ sink implemented in `bigquery_file_loads.py` does not handle these limitations well. Improvements need to be made to the miplementation, to:

• Decide to use temp_tables dynamically at pipeline execution
• Add code to determine when a load job to a single destination needs to be partitioned into multiple jobs.
• When this happens, then we definitely need to use temp_tables, in case one of the two load jobs fails, and the pipeline is rerun.

Tanay, would you be able to look at this?

Issue Links

• links to: GitHub Pull Request #9242