[BEAM-10260] Nested iterable in state cache exceeds Python recursion limit

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.23.0
• Component/s: sdk-py-harness
• Assignee: Maximilian Michels
• Reporter: Maximilian Michels
• Created: 15/Jun/20 12:51
• Updated: 16/Jun/20 16:25
• Resolved: 16/Jun/20 14:27

Description

With state caching enabled, the changes in BEAM-8298 lead to an unbounded nesting of state iterables when appending to a bag state, quickly resulting in an exception:

events = list(event_bag.read())  
File "/code/venvs/venv/lib/python3.6/site-packages/apache_beam/runners/worker/bundle_processor.py", line 488, in __iter__    for elem in self.first:  
File "/code/venvs/venv/lib/python3.6/site-packages/apache_beam/runners/worker/statecache.py", line 186, in __iter__    for item in value:  
File "/code/venvs/venv/lib/python3.6/site-packages/apache_beam/runners/worker/statecache.py", line 186, in __iter__    for item in value:  
File "/code/venvs/venv/lib/python3.6/site-packages/apache_beam/runners/worker/statecache.py", line 186, in __iter__    for item in value:  [Previous line repeated 977 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object 

Issue Links

• is caused by: BEAM-8298 Implement state caching for side inputs for Python
• links to: GitHub Pull Request #12012