[ARROW-369] [Python] Add ability to convert multiple record batches at once to pandas

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.2.0
• Component/s: Python
• Labels: newbie
• External issue URL: https://github.com/apache/arrow/issues/15932
• Assignee: Bryan Cutler
• Reporter: Wes McKinney
• Created: 07/Nov/16 21:49
• Updated: 11/Jan/23 10:08
• Resolved: 02/Dec/16 22:53

Description

Instead of only being able to only convert single single record batches and tables that consist only of single ColumnChunks, we should also support the construction of Pandas DataFrames from multiple RecordBatches. In the most simple way, we would convert each batch to a Pandas DataFrame and then concat them all together. A second (and preferred) implementation would extend the C++ function ConvertColumnToPandas in python/src/pyarrow/adapters/pandas.* to work on chunked columns.

Issue Links

• relates to: ARROW-396 Python: Add pyarrow.schema.Schema.equals