Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12514

BigQueryIO - ReadFromBigQuery can not get table reference from RuntimeValueProvider

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Triage Needed
    • Priority: P2
    • Resolution: Unresolved
    • Affects Version/s: 2.26.0, 2.27.0, 2.28.0, 2.29.0, 2.30.0
    • Fix Version/s: None
    • Component/s: io-py-gcp

      Description

      After this change, the table reference couldn't be parsed correctly when ReadFromBigQuery is taking a RuntimeValueProvider as value of table.

      This bug is included since v2.26.0

       

      Code to reproduce:

      import apache_beam as beam
      from apache_beam.options.pipeline_options import PipelineOptions
      
      
      class MyOptions(PipelineOptions):
          @classmethod
          def _add_argparse_args(cls, parser):
              parser.add_value_provider_argument(
                  "--input",
                  type=str,
                  default='test-project:test_dataset.test_table',
              )
      
      
      pipeline_options = PipelineOptions()
      with beam.Pipeline(options=pipeline_options) as p:
          my_options = pipeline_options.view_as(MyOptions)
          lines = p | "read" >> beam.io.ReadFromBigQuery(table=my_options.input)
          row_count = lines | "count" >> beam.combiners.Count.Globally()
          row_count | beam.Map(print)
      
          p.run().wait_until_finish()
      

      Error message will be:

      Traceback (most recent call last):
        ..
        ..
        File "....../venv/lib/python3.8/site-packages/apache_beam/io/gcp/bigquery.py", line 722, in estimate_size
          if not table_ref.projectId:
      AttributeError: 'RuntimeValueProvider' object has no attribute 'projectId' [while running 'read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction']
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                chutium Teng Qiu
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 40m
                  1h 40m