[ARROW-15517] [R] Use WriteNode in write_dataset()

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.0.0
• Component/s: R
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/30990
• Assignee: Neal Richardson
• Reporter: Neal Richardson
• Created: 01/Feb/22 17:20
• Updated: 11/Jan/23 11:37
• Resolved: 19/Apr/22 21:41

Description

Currently, write_dataset uses the Scanner interface, which can't handle everything that the ExecPlan does. So if your arrow_dplyr_query contains things like aggregations or (more importantly) joins, you have to materialize the Table in memory before you can write to disk. The WriteNode added in ARROW-13542 is a special sink node that can be put at the end of an ExecPlan, so data should be able to stream to disk in more cases, and will benefit from future improvements to ExecPlan memory usage and spillover.

Issue Links

• is duplicated by: ARROW-14266 [R] Use WriteNode to write queries
• links to: GitHub Pull Request #12316