[ARROW-16703] [R] Refactor map_batches() so it can stream results

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 8.0.0
• Fix Version/s: 9.0.0
• Component/s: R
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/32046

Language

R

• Assignee: Dewey Dunnington
• Reporter: Will Jones
• Created: 31/May/22 21:15
• Updated: 11/Jan/23 11:45
• Resolved: 23/Jul/22 12:09

Description

As part of ARROW-15271, map_batches() was modified to return a RecordBatchReader, but the implementation collects all results as a list of record batches and then converts that to a reader. In theory, if we push the implementation down to C++, we should be able to make a proper streaming RBR.

We won't know the schema ahead of time. We could optionally accept it, which would allow the function to be lazy. Or we could eagerly evaluate just the first batch to determine the schema.

Issue Links

• links to: GitHub Pull Request #13650