[ARROW-1783] [Python] Convert SerializedPyObject to/from sequence of component buffers with minimal memory allocation / copying

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/17781
• Assignee: Wes McKinney
• Reporter: Wes McKinney
• Created: 09/Nov/17 17:33
• Updated: 11/Jan/23 07:16
• Resolved: 27/Nov/17 21:15

Description

See discussion on Dask org:

https://github.com/dask/distributed/pull/931

It would be valuable for downstream users to compute the serialized payload as a sequence of memoryview-compatible objects without having to allocate new memory on write. This means that the component tensor messages must have their metadata and bodies in separate buffers. This will require a bit of work internally reassemble the object from a collection of pyarrow.Buffer objects

see also ARROW-1509

Issue Links

• is related to: ARROW-1784 [Python] Read and write pandas.DataFrame in pyarrow.serialize by decomposing the BlockManager rather than coercing to Arrow format
• relates to: ARROW-1509 [Python] Write serialized object as a stream of encapsulated IPC messages
• links to: GitHub Pull Request #1362