[ARROW-10573] Zero-Copy Pandas Dataframe Reads

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.0
• Component/s: C++, Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/26537
• Assignee: Unassigned
• Reporter: Nick White
• Created: 12/Nov/20 22:56
• Updated: 11/Jan/23 08:14

Description

Arrow should be able to expose an IPC message as a Pandas dataframe without copying the data. This involves fixing:

• the alignment of the IPC message, so all columns of the same type are in contiguous buffers and so can be exposed as the 2D arrays Pandas' block manager expects.
• the `arrow_to_pandas.cc` code to make the UX work transparently.

Issue Links

• links to: GitHub Pull Request #8644