[BEAM-9549] Flaky portableWordCountBatch and portableWordCountStreaming tests

• Type: Test
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: test-failures
• Assignee: Unassigned
• Reporter: Ning
• Created: 18/Mar/20 21:49
• Updated: 04/Jun/22 15:40

Description

The tests :sdks:python:test-suites:portable:py2:portableWordCountBatch and :sdks:python:test-suites:portable:py2:portableWordCountStreaming are flaky, sometimes throws grpc errrors.

Stacktrace

In text:

INFO:root:Using Python SDK docker image: apache/beam_python2.7_sdk:2.21.0.dev. If the image is not available at local, we will try to pull from hub.docker.comINFO:apache_beam.runners.portability.fn_api_runner_transforms:==================== <function lift_combiners at 0x7fdd4225b050> ====================INFO:apache_beam.utils.subprocess_server:Starting service with ['docker' 'run' '-v' '/usr/bin/docker:/bin/docker' '-v' '/var/run/docker.sock:/var/run/docker.sock' '--network=host' 'apache/beam_flink1.9_job_server:latest' '--job-host' 'localhost' '--job-port' '58753' '--artifact-port' '60175' '--expansion-port' '33067']INFO:apache_beam.utils.subprocess_server:[main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - ArtifactStagingService started on localhost:60175INFO:apache_beam.utils.subprocess_server:[main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java ExpansionService started on localhost:33067INFO:apache_beam.utils.subprocess_server:[main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - JobService started on localhost:58753ERROR:grpc._common:Exception deserializing message!Traceback (most recent call last):  File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py", line 84, in _transform    return transformer(message)DecodeError: Error parsing messageTraceback (most recent call last):  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main    "__main__", fname, loader, pkg_name)  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code    exec code in run_globals  File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/lib/python2.7/site-packages/apache_beam/examples/wordcount.py", line 142, in <module>    run()  File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/lib/python2.7/site-packages/apache_beam/examples/wordcount.py", line 121, in run    result = p.run()  File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 495, in run    self._options).run(False)  File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 508, in run    return self.runner.run_pipeline(self, self._options)  File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 401, in run_pipeline    job_service_handle.submit(proto_pipeline)  File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 102, in submit    prepare_response = self.prepare(proto_pipeline)  File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 179, in prepare    timeout=self.timeout)  File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py", line 826, in __call__    return _end_unary_response_blocking(state, call, False, None)  File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking    raise _InactiveRpcError(state)grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:	status = StatusCode.INTERNAL	details = "Exception deserializing response!"	debug_error_string = "None">

A gradle scan example.