[ARROW-15040] [R] Enable write_csv_arrow to take Dataset or arrow_dplyr_query as input

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.0.0
• Component/s: R
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/30557
• Assignee: Nicola Crane
• Reporter: Nicola Crane
• Created: 09/Dec/21 08:10
• Updated: 11/Jan/23 08:44
• Resolved: 01/Mar/22 18:08

Description

Currently, this code fails:

dataset <- open_dataset("some/folder/with/parquet/files")
write_csv_arrow(dataset, sink = "dataset.csv")

with this error message:

Error: x must be an object of class 'data.frame', 'RecordBatch', or 'Table', not 'FileSystemDataset'.

In ARROW-14741, support was added for reading from a RecordBatchReader, so we should be able to now extend write_csv_arrow() to allow this behaviour.

 

Note: We would need to make sure whatever write_csv(record_batch_reader) function can take a filesystem= argument

Issue Links

• depends upon: ARROW-14741 [C++] Allow CSV Writer to take a RecordBatchReader as input
• is blocked by: ARROW-15128 [C++] segfault when writing CSV from RecordBatchReader
• is duplicated by: ARROW-15104 write_parquet() / write_csv_arrow() cannot stream a dataset object back to S3
• relates to: ARROW-15271 [R] Refactor do_exec_plan to return a RecordBatchReader
• links to: GitHub Pull Request #11971