[BEAM-1440] Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK

• Type: New Feature
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.19.0
• Component/s: sdk-py-core
• Assignee: Kamil Wasilewski
• Reporter: Chamikara Madhusanka Jayalath
• Created: 08/Feb/17 23:52
• Updated: 13/Apr/23 11:00
• Resolved: 02/Jan/20 08:57

Description

Currently we have a BigQuery native source for Python SDK [1].
This can only be used by Dataflow runner.

We should implement a Beam BigQuery source that implements iobase.BoundedSource [2] interface so that other runners that try to use Python SDK can read from BigQuery as well. Java SDK already has a Beam BigQuery source [3].

[1] https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py
[2] https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py#L70
[3] https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1189

Issue Links

• blocks: BEAM-3097 Allow BigQuerySource to take a ValueProvider as a table input.
• causes: BEAM-8528 BigQuery bounded source does not work on DirectRunner
• causes: BEAM-8910 Use AVRO instead of JSON in BigQuery bounded source.
• links to: GitHub Pull Request #9772
• links to: SO: Dataflow template job is not taking input parameters
• links to: SO: Dynamic bigquery query in dataflow template