[BEAM-10872] Value Provider functionality broken in python sdk

• Type: Bug
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Affects Version/s: 2.22.0
• Component/s: runner-dataflow
• Environment: requirements.txt can be found here: https://gist.github.com/jay-karimi/20524163266e7c90b2fc9b7a6b401e43
• Flags: Important
• Assignee: Unassigned
• Reporter: Jay K
• Created: 10/Sep/20 17:12
• Updated: 04/Jun/22 18:22

Description

Whether using a custom function or an IO connector (test with `WriteToBigQuery`), the dataflow job complains that the runtime value provider `get()` is being called from a non-runtime context:

import argparse
import logging

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions


class ExampleDoF(beam.DoFn):
    def __init__(self, value_provider):
        self.value_provider = value_provider

    def process(self, el):
        logging.info(f'el: {el}')
        logging.info(f'value provider: {self.value_provider.get()}')
        yield el


class UserOptions(PipelineOptions):
    @classmethod
    def _add_argparse_args(cls, parser):
        parser.add_value_provider_argument('--value_provider', type=int)

parser = argparse.ArgumentParser()
known_args, pipeline_args = parser.parse_known_args()
pipeline_options = PipelineOptions(pipeline_args)
user_options = pipeline_options.view_as(UserOptions)
with beam.Pipeline(options=pipeline_options) as pipeline:
    results = (
            pipeline
            | beam.Create(['element'])
            | beam.ParDo(ExampleDoF(user_options.value_provider))
    )