[BEAM-9907] apache_beam.transforms.external_test.ExternalTransformTest.test_nested flaky

• Type: Test
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.22.0
• Component/s: sdk-py-core
• Assignee: Brian Hulette
• Reporter: Ning
• Created: 06/May/20 19:06
• Updated: 16/May/20 14:01
• Resolved: 08/May/20 16:54

Description

Example test failures:

https://builds.apache.org/job/beam_PreCommit_Python_Commit/12682/
https://builds.apache.org/job/beam_PreCommit_Python_Commit/12684/

A stacktrace

apache_beam.transforms.external_test.ExternalTransformTest.test_nested (from py37-cloud)

Failing for the past 1 build (Since Failed#12682 )
Took 54 ms.
Error Message
google.protobuf.json_format.ParseError: Unexpected type for Value message.
Stacktrace
self = <apache_beam.transforms.external_test.ExternalTransformTest testMethod=test_nested>

    def test_nested(self):
      with beam.Pipeline() as p:
>       assert_that(p | FibTransform(6), equal_to([8]))

apache_beam/transforms/external_test.py:250: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
apache_beam/transforms/ptransform.py:562: in __ror__
    result = p.apply(self, pvalueish, label)
apache_beam/pipeline.py:651: in apply
    pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
apache_beam/runners/runner.py:198: in apply
    return m(transform, input, options)
apache_beam/runners/runner.py:228: in apply_PTransform
    return transform.expand(input)
apache_beam/runners/portability/expansion_service_test.py:257: in expand
    expansion_service.ExpansionServiceServicer())
apache_beam/pvalue.py:140: in __or__
    return self.pipeline.apply(ptransform, self)
apache_beam/pipeline.py:598: in apply
    transform.transform, pvalueish, label or transform.label)
apache_beam/pipeline.py:608: in apply
    return self.apply(transform, pvalueish)
apache_beam/pipeline.py:651: in apply
    pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
apache_beam/runners/runner.py:198: in apply
    return m(transform, input, options)
apache_beam/runners/runner.py:228: in apply_PTransform
    return transform.expand(input)
apache_beam/transforms/external.py:322: in expand
    pipeline_options=job_utils.pipeline_options_dict_to_struct(options))
apache_beam/runners/job/utils.py:38: in pipeline_options_dict_to_struct
    v in options.items() if v is not None
apache_beam/runners/job/utils.py:44: in dict_to_struct
    return json_format.ParseDict(dict_obj, struct_pb2.Struct())
target/.tox-py37-cloud/py37-cloud/lib/python3.7/site-packages/google/protobuf/json_format.py:450: in ParseDict
    parser.ConvertMessage(js_dict, message)
target/.tox-py37-cloud/py37-cloud/lib/python3.7/site-packages/google/protobuf/json_format.py:479: in ConvertMessage
    methodcaller(_WKTJSONMETHODS[full_name][1], value, message)(self)
target/.tox-py37-cloud/py37-cloud/lib/python3.7/site-packages/google/protobuf/json_format.py:667: in _ConvertStructMessage
    self._ConvertValueMessage(value[key], message.fields[key])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <google.protobuf.json_format._Parser object at 0x7f35a4c62e48>
value = <apache_beam.options.value_provider.RuntimeValueProvider object at 0x7f35a4c00390>
message = 

    def _ConvertValueMessage(self, value, message):
      """Convert a JSON representation into Value message."""
      if isinstance(value, dict):
        self._ConvertStructMessage(value, message.struct_value)
      elif isinstance(value, list):
        self. _ConvertListValueMessage(value, message.list_value)
      elif value is None:
        message.null_value = 0
      elif isinstance(value, bool):
        message.bool_value = value
      elif isinstance(value, six.string_types):
        message.string_value = value
      elif isinstance(value, _INT_OR_FLOAT):
        message.number_value = value
      else:
>       raise ParseError('Unexpected type for Value message.')
E       google.protobuf.json_format.ParseError: Unexpected type for Value message.

target/.tox-py37-cloud/py37-cloud/lib/python3.7/site-packages/google/protobuf/json_format.py:647: ParseError

Issue Links

• relates to: BEAM-9449 Consider passing pipeline options for expansion service.
• links to: GitHub Pull Request #11631