[ARROW-823] [Python] Devise a means to serialize arrays of arbitrary Python objects in Arrow IPC messages

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Won't Do
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/16431
• Assignee: Unassigned
• Reporter: Wes McKinney
• Created: 14/Apr/17 19:57
• Updated: 11/Jan/23 07:11
• Resolved: 04/Aug/21 08:23

Description

Practically speaking, this would involve a "custom" logical type that is "pyobject", represented physically as an array of 64-bit pointers. On serialization, this would need to be converted to a BinaryArray containing pickled objects as binary values

At the moment, we don't yet have the machinery to deal with "custom" types where the in-memory representation is different from the on-wire representation. This would be a useful use case to work through the design issues

Interestingly, if done properly, this would enable other Arrow implementations to manipulate (filter, etc.) serialized Python objects as binary blobs.