[ARROW-504] [Python] Add adapter to write pandas.DataFrame in user-selected chunk size to streaming format

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/16145
• Assignee: Wes McKinney
• Reporter: Wes McKinney
• Created: 21/Jan/17 19:11
• Updated: 11/Jan/23 07:09
• Resolved: 27/Nov/17 09:28

Description

While we can convert a pandas.DataFrame to a single (arbitrarily large) arrow::RecordBatch, it is not easy to create multiple small record batches – we could do so in a streaming fashion and immediately write them into an arrow::io::OutputStream.

Issue Links

• relates to: ARROW-232 C++/Parquet: Support writing chunked arrays as part of a table