[ARROW-12083] [R] schema use in open_dataset

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 5.0.0
• Component/s: R
• Labels: pull-request-available
• Environment: Windows
• External issue URL: https://github.com/apache/arrow/issues/27908
• Assignee: David Li
• Reporter: Shaun Nielsen
• Created: 25/Mar/21 01:20
• Updated: 11/Jan/23 08:24
• Resolved: 10/May/21 16:48

Description

I have a directory of split .csvs that I'm importing with open_dataset(). Between files, a column is imported as either int64 (e.g. -2) and the other string (1986CD), and this throws an error when unify_schemas = T

{{ arrow::open_dataset('./split-csvs/nswcr/', format = 'csv', unify_schemas = T)}}

Error: Invalid: Unable to merge: Field SEIFACalcMethod has incompatible types: int64 vs string

If I use the schema parameter, and only want to specify this column, I only am able to import this column

arrow::open_dataset('./split-csvs/nswcr/', format = 'csv', schema = schema(SEIFACalcMethod = string()))

{{ }}
FileSystemDataset with 45 csv files
SEIFACalcMethod: string

I was expecting that could set the class of a select few columns, while the rest would be imported as-is. Similar to readr::read_csv(col_types = cols()) approach.

Not sure if this is expected behaviour, a bug, or a possible avenue for improvement. I've tagged this as the latter. 

 

Issue Links

• links to: GitHub Pull Request #10284