[ARROW-15823] [C++][Python] Add a method to convert a Table to a RecordBatchReader

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.0.0
• Component/s: C++, Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/31264
• Assignee: Sanjiban Sengupta
• Reporter: Weston Pace
• Created: 02/Mar/22 04:24
• Updated: 11/Jan/23 11:39
• Resolved: 16/Mar/22 19:33

Description

The RecordBatchReader interface is a key abstraction for the C stream API. There is no great way today to convert a Table (or an iterable of batches) to a RecordBatchReader.

DuckDb is currently doing this by wrapping the Table in an in-memory dataset and consuming it with a scanner but this has considerable overhead (especially if the Table has many batches).

Alternatively, we could add bridge methods to export a table to a stream but I think this approach would be simpler.

Issue Links

• links to: GitHub Pull Request #12621