[BEAM-12514] BigQueryIO - ReadFromBigQuery can not get table reference from RuntimeValueProvider

• Type: Bug
• Status: Triage Needed
• Priority: P3
• Resolution: Fixed
• Affects Version/s: 2.26.0, 2.27.0, 2.28.0, 2.29.0, 2.30.0, 2.31.0
• Fix Version/s: 2.32.0
• Component/s: io-py-gcp
• Labels: GCP pull-request-available
• Assignee: Teng Qiu
• Reporter: Teng Qiu
• Created: 18/Jun/21 17:24
• Updated: 13/Apr/23 11:05
• Resolved: 15/Oct/21 20:47

Description

After this change, the table reference couldn't be parsed correctly when ReadFromBigQuery is taking a RuntimeValueProvider as value of table.

This bug is included since v2.26.0

 

Code to reproduce:

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions


class MyOptions(PipelineOptions):
    @classmethod
    def _add_argparse_args(cls, parser):
        parser.add_value_provider_argument(
            "--input",
            type=str,
            default='test-project:test_dataset.test_table',
        )


pipeline_options = PipelineOptions()
with beam.Pipeline(options=pipeline_options) as p:
    my_options = pipeline_options.view_as(MyOptions)
    lines = p | "read" >> beam.io.ReadFromBigQuery(table=my_options.input)
    row_count = lines | "count" >> beam.combiners.Count.Globally()
    row_count | beam.Map(print)

    p.run().wait_until_finish()

Error message will be:

Traceback (most recent call last):
  ..
  ..
  File "....../venv/lib/python3.8/site-packages/apache_beam/io/gcp/bigquery.py", line 722, in estimate_size
    if not table_ref.projectId:
AttributeError: 'RuntimeValueProvider' object has no attribute 'projectId' [while running 'read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction']

Issue Links

• links to: GitHub Pull Request #15040