[ARROW-1784] [Python] Read and write pandas.DataFrame in pyarrow.serialize by decomposing the BlockManager rather than coercing to Arrow format

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/17782
• Assignee: Wes McKinney
• Reporter: Wes McKinney
• Created: 09/Nov/17 17:35
• Updated: 11/Jan/23 07:16
• Resolved: 06/Dec/17 19:10

Description

See discussion in https://github.com/dask/distributed/pull/931

This will permit zero-copy reads for DataFrames not containing Python objects. In the event of an ObjectBlock these arrays will not be worse than pickle to reconstruct on the receiving side

Issue Links

• is related to: ARROW-1887 [Python] More efficient serialization of pandas Index types in custom serialization from ARROW-1784
• relates to: ARROW-1783 [Python] Convert SerializedPyObject to/from sequence of component buffers with minimal memory allocation / copying
• relates to: ARROW-1854 [Python] Improve performance of serializing object dtype ndarrays
• links to: GitHub Pull Request #1390