[ARROW-7018] [R] Non-UTF-8 data in Arrow <--> R conversion

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.15.0
• Fix Version/s: 1.0.0
• Component/s: R
• Labels: pull-request-available
• Environment: I'm running R on Windows 10
• External issue URL: https://github.com/apache/arrow/issues/23331
• Assignee: Neal Richardson
• Reporter: Vidar Ingason
• Created: 29/Oct/19 08:08
• Updated: 11/Jan/23 07:51
• Resolved: 26/Jun/20 15:19

Description

Hello.
I'm new to the arrow package in R and I'm having a trouble regarding special characters (Icelandic). I have a large data set and everything is fine until I write the file to disk and read it in again (i.e. I use write_parquet() and then read_parquet()). When I read the data back in to R special characters turn into question mark. I.e. Veitingastaðir becomes Veitingasta�ir.

This does not happen when I use .csv.

Is there anything I can do when I write the .parquet file to disk or when I read it in to prevent this?

Issue Links

• is related to: ARROW-7288 [C++][R] read_parquet() freezes on Windows with Japanese locale
• is related to: ARROW-9163 [C++] Add methods to StringArray, LargeStringArray, to validate whether its values are all UTF-8
• links to: GitHub Pull Request #7527