[BEAM-9975] PortableRunnerTest flake "ParseError: Unexpected type for Value message."

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Fix Version/s: 2.24.0
• Component/s: sdk-py-core
• Labels: beam-fixit flake
• Assignee: Brian Hulette
• Reporter: Brian Hulette
• Created: 12/May/20 20:52
• Updated: 10/Jul/20 17:13
• Resolved: 10/Jul/20 17:13

Description

Error looks similar to the one in BEAM-9907. Example from https://builds.apache.org/job/beam_PreCommit_Python_Cron/2732

apache_beam/runners/portability/fn_api_runner/fn_runner_test.py:569: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
apache_beam/pipeline.py:550: in __exit__
    self.run().wait_until_finish()
apache_beam/pipeline.py:529: in run
    return self.runner.run_pipeline(self, self._options)
apache_beam/runners/portability/portable_runner.py:426: in run_pipeline
    job_service_handle.submit(proto_pipeline)
apache_beam/runners/portability/portable_runner.py:107: in submit
    prepare_response = self.prepare(proto_pipeline)
apache_beam/runners/portability/portable_runner.py:184: in prepare
    pipeline_options=self.get_pipeline_options()),
apache_beam/runners/portability/portable_runner.py:174: in get_pipeline_options
    return job_utils.dict_to_struct(p_options)
apache_beam/runners/job/utils.py:33: in dict_to_struct
    return json_format.ParseDict(dict_obj, struct_pb2.Struct())
target/.tox-py36-cython/py36-cython/lib/python3.6/site-packages/google/protobuf/json_format.py:450: in ParseDict
    parser.ConvertMessage(js_dict, message)
target/.tox-py36-cython/py36-cython/lib/python3.6/site-packages/google/protobuf/json_format.py:479: in ConvertMessage
    methodcaller(_WKTJSONMETHODS[full_name][1], value, message)(self)
target/.tox-py36-cython/py36-cython/lib/python3.6/site-packages/google/protobuf/json_format.py:667: in _ConvertStructMessage
    self._ConvertValueMessage(value[key], message.fields[key])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <google.protobuf.json_format._Parser object at 0x7f69eb5b59e8>
value = <apache_beam.options.value_provider.RuntimeValueProvider object at 0x7f69eb7b3ac8>
message = 

    def _ConvertValueMessage(self, value, message):
      """Convert a JSON representation into Value message."""
      if isinstance(value, dict):
        self._ConvertStructMessage(value, message.struct_value)
      elif isinstance(value, list):
        self. _ConvertListValueMessage(value, message.list_value)
      elif value is None:
        message.null_value = 0
      elif isinstance(value, bool):
        message.bool_value = value
      elif isinstance(value, six.string_types):
        message.string_value = value
      elif isinstance(value, _INT_OR_FLOAT):
        message.number_value = value
      else:
>       raise ParseError('Unexpected type for Value message.')
E       google.protobuf.json_format.ParseError: Unexpected type for Value message.

Issue Links

• is caused by: BEAM-10006 PipelineOptions can pick up definitions from unrelated tests
• is caused by: BEAM-10007 PortableRunner doesn't handle ValueProvider instances when converting pipeline options
• links to: GitHub Pull Request #11696