[ARROW-11371] [Python] Serializing deeply nested structures raises recursion error

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 1.0.1
• Component/s: Python
• Labels: pyarrow-serialization
• Environment: $ python --version ; Python 3.6.12 ; $ pip list | grep pyarrow ; pyarrow 1.0.1
• External issue URL: https://github.com/apache/arrow/issues/27264
• Assignee: Unassigned
• Reporter: Alexander Dmitriev
• Created: 25/Jan/21 10:42
• Updated: 11/Jan/23 08:19
• Resolved: 17/Feb/21 16:01

Description

When serializing deeply nested python structures such as:

data = [[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[None]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]

or 

data = OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', None)]))]))]))]))]))]))]))]))])

pyarrow raises the following exception:

>>> buffer = pyarrow.serialize(data).to_buffer()
Traceback (most recent call last):
  File "pyarrow_recursion.py", line 11, in try_recursion
    buffer = pyarrow.serialize(data).to_buffer()
  File "pyarrow/serialization.pxi", line 285, in pyarrow.lib.SerializedPyObject.to_buffer
  File "pyarrow/serialization.pxi", line 248, in pyarrow.lib.SerializedPyObject.total_bytes.__get__
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Max recursion depth reached

While this not the prime use case of pyarrow.serialize, it would be nice to have a much larger recursion limit here, to not run into this issue with auto-generated structures quite so often.

An example script for reproducing the error can be found in the attachments.