[ARROW-18352] [R] Datasets API interface improvements

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: R
• External issue URL: https://github.com/apache/arrow/issues/33520
• Assignee: Unassigned
• Reporter: Nicola Crane
• Created: 17/Nov/22 18:07
• Updated: 11/Jan/23 11:59

Description

Umbrella ticket for improvements for our interface to the datasets API, and making the experience more consistent between open_dataset() and the read_*() functions.

The current parameters which are supported in read_delim_arrow() but not in open_dataset are:

• file
• col_names
• col_select
• na
• quoted_na
• parse_options/convert_options/read_options*
• as_data_frame

Subtasks 2, 5, 6, 7, 8, and 10 below allow us to support all of the read_csv_arrow() options in open_dataset() for CSVs, or give helpful error messages when options aren't supported.

Sub-Tasks

• [R] Better document the CSV read/parse/convert options we can use with open_dataset(): Open
• [R] Allows user to specify string to be used for missing data when writing CSV dataset: Resolved
• [R] Update function examples to show how to open CSV dataset with partitioning and a schema: Open
• [R] passing a schema calls open_dataset to fail on hive-partitioned csv files: Open
• [R] CsvConvertOptions include_columns and col_select should give better error message when used in open_dataset: Open
• [R] Allow user to supply col_names argument when reading in a CSV dataset: Resolved
• [R] support the quoted_na argument in open_dataset for CSVs by mapping it to CSVConvertOptions$strings_can_be_null: Closed
• [R] Handle as_data_frame argument if passed into open_dataset for CSVs: Open
• [R] support parse_options, read_options, convert_options in open_dataset to mirror read_csv_arrow: Closed
• [R] Update read_csv_arrow and open_dataset parse_options, read_options, and convert_options to take lists: In Progress
• [R] Implement new function open_dataset_csv with signature more closely matching read_csv_arrow: Open
• [R] Misleading error message if opening CSV dataset with invalid file in directory: Open
• [R] Improve error message when providing a mix of readr and Arrow options: Open