[BEAM-1894] Race conditions in python direct runner eager mode

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: sdk-py-core
• Assignee: Udi Meiri
• Reporter: Vikas Kedigehalli
• Created: 05/Apr/17 23:53
• Updated: 18/Jun/20 22:26
• Resolved: 18/Jun/20 22:26

Description

test_eager_pipeline (https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pipeline_test.py#L283) fails with the following error:
ERROR: test_eager_pipeline (apache_beam.pipeline_test.PipelineTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/pipeline_test.py", line 285, in test_eager_pipeline
self.assertEqual([1, 4, 9], p | Create([1, 2, 3]) | Map(lambda x: x*x))
File "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/transforms/ptransform.py", line 387, in _ror_
p.run().wait_until_finish()
File "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/pipeline.py", line 160, in run
self.to_runner_api(), self.runner, self.options).run(False)
File "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/pipeline.py", line 169, in run
return self.runner.run(self)
File "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/direct_runner.py", line 99, in run
result.wait_until_finish()
File "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/direct_runner.py", line 166, in wait_until_finish
self._executor.await_completion()
File "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/executor.py", line 336, in await_completion
self._executor.await_completion()
File "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/executor.py", line 308, in _call_
uncommitted_bundle.get_elements_iterable())
File "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/evaluation_context.py", line 176, in append_to_cache
self._cache.append(applied_ptransform, tag, elements)
File "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/direct_runner.py", line 138, in append
self._cache[(applied_ptransform, tag)].extend(elements)
TypeError: 'NoneType' object has no attribute '_getitem_'

This is triggered when Create is changed to a custom source.

Issue Links

• links to: GitHub Pull Request #11242