[BEAM-9963] Flink streaming load tests fail with TypeError

• Type: Test
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: build-system, runner-flink
• Assignee: Maximilian Michels
• Reporter: Maximilian Michels
• Created: 12/May/20 09:50
• Updated: 16/May/20 14:18
• Resolved: 12/May/20 11:27

Description

The newly added load tests now fail on the latest master.

https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Streaming_PR/39/console

16:00:38 INFO:apache_beam.runners.portability.portable_runner:Job state changed to FAILED
16:00:38 Traceback (most recent call last):
16:00:38   File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
16:00:38     "__main__", mod_spec)
16:00:38   File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
16:00:38     exec(code, run_globals)
16:00:38   File "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_ParDo_Flink_Streaming_PR/src/sdks/python/apache_beam/testing/load_tests/pardo_test.py", line 225, in <module>
16:00:38     ParDoTest().run()
16:00:38   File "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_ParDo_Flink_Streaming_PR/src/sdks/python/apache_beam/testing/load_tests/load_test.py", line 115, in run
16:00:38     self.result.wait_until_finish(duration=self.timeout_ms)
16:00:38   File "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_ParDo_Flink_Streaming_PR/src/sdks/python/apache_beam/runners/portability/portable_runner.py", line 583, in wait_until_finish
16:00:38     raise self._runtime_exception
16:00:38 RuntimeError: Pipeline load-tests-python-flink-streaming-pardo-5-0508133349_3afe3b8e-23f7-474b-af08-4ee7feb3adfa failed in state FAILED: java.lang.RuntimeException: Error received from SDK harness for instruction 2: Traceback (most recent call last):
16:00:38   File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 245, in _execute
16:00:38     response = task()
16:00:38   File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 302, in <lambda>
16:00:38     lambda: self.create_worker().do_instruction(request), request)
16:00:38   File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 471, in do_instruction
16:00:38     getattr(request, request_type), request.instruction_id)
16:00:38   File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 506, in process_bundle
16:00:38     bundle_processor.process_bundle(instruction_id))
16:00:38   File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 918, in process_bundle
16:00:38     op.finish()
16:00:38   File "apache_beam/runners/worker/operations.py", line 697, in apache_beam.runners.worker.operations.DoOperation.finish
16:00:38   File "apache_beam/runners/worker/operations.py", line 699, in apache_beam.runners.worker.operations.DoOperation.finish
16:00:38   File "apache_beam/runners/worker/operations.py", line 702, in apache_beam.runners.worker.operations.DoOperation.finish
16:00:38   File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 709, in commit
16:00:38     state.commit()
16:00:38   File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 414, in commit
16:00:38     self._underlying_bag_state.commit()
16:00:38   File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 481, in commit
16:00:38     is_cached=True)
16:00:38   File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 951, in extend
16:00:38     coder.encode_to_stream(element, out, True)
16:00:38   File "apache_beam/coders/coder_impl.py", line 690, in apache_beam.coders.coder_impl.VarIntCoderImpl.encode_to_stream
16:00:38   File "apache_beam/coders/coder_impl.py", line 692, in apache_beam.coders.coder_impl.VarIntCoderImpl.encode_to_stream
16:00:38 TypeError: an integer is required
16:00:38 

Cron fails as well: https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Streaming/5/console

Issue Links

• relates to: BEAM-8742 Add stateful processing to ParDo load test