[ARROW-3587] [Python] Efficient serialization for Arrow Objects (array, table, tensor, etc)

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.12.0
• Component/s: C++, C++ - Plasma, Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/19899
• Assignee: Siyuan Zhuang
• Reporter: Siyuan Zhuang
• Created: 22/Oct/18 18:59
• Updated: 11/Jan/23 07:28
• Resolved: 08/Nov/18 05:55

Description

Currently, Arrow seems to have poor serialization support for its own objects.

For example,
 

import pyarrow 
arr = pyarrow.array([1, 2, 3, 4]) 
pyarrow.serialize(arr)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyarrow/serialization.pxi", line 337, in pyarrow.lib.serialize
File "pyarrow/serialization.pxi", line 136, in pyarrow.lib.SerializationContext._serialize_callback
pyarrow.lib.SerializationCallbackError: pyarrow does not know how to serialize objects of type <class 'pyarrow.lib.Int64Array'>.

I am working Ray & modin project, using plasma to store Arrow objects. Lack of direct serialization support harms the performance, so I would like to push a PR to fix this problem.
I wonder if it is welcome or is there someone else doing it?

Issue Links

• links to: GitHub Pull Request #2832