[ARROW-2660] [Python] Experiment with zero-copy pickling

• Type: Wish
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.10.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/19054
• Assignee: Antoine Pitrou
• Reporter: Antoine Pitrou
• Created: 01/Jun/18 08:42
• Updated: 11/Jan/23 07:22
• Resolved: 30/Jul/18 13:31

Description

PEP 574 has an implementation ready and a PyPI-available backport (at https://pypi.org/project/pickle5/ ). Adding experimental support for it would allow for zero-copy pickling of Arrow arrays, columns, etc.

I think it mainly involves implementing reduce_ex on the Buffer class, as described in https://www.python.org/dev/peps/pep-0574/#producer-api

In addition, the consumer API added by PEP 574 could be used in Arrow's serialization layer, to avoid or minimize copies when serializing foreign objects.

Issue Links

• links to: GitHub Pull Request #2161