[ARROW-6582] [R] Arrow to R fails with embedded nuls in strings

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.1
• Fix Version/s: 3.0.0
• Component/s: R
• Labels: pull-request-available
• Environment: Windows 10 ; R 3.4.4
• External issue URL: https://github.com/apache/arrow/issues/22939
• Assignee: Neal Richardson
• Reporter: John Cassil
• Created: 17/Sep/19 14:32
• Updated: 11/Jan/23 07:48
• Resolved: 04/Jan/21 20:48

Description

Apologies if this issue isn't categorized or documented appropriately. Please be gentle!

As a heavy R user that normally interacts with parquet files using SparklyR, I have recently decided to try to use arrow::read_parquet() on a few parquet files that were on my local machine rather than in hadoop. I was not able to proceed after several various attempts due to embedded nuls. For example:

try(

{df <- read_parquet('out_2019-09_data_1.snappy.parquet') }

)
Error in Table__to_dataframe(x, use_threads = option_use_threads()) :
embedded nul in string: 'INSTALL BOTH LEFT FRONT AND RIGHT FRONT TORQUE ARMS\0 ARMS'

Is there a solution to this?

I have also hit roadblocks with embedded nuls in the past with csvs using data.table::fread(), but readr::read_delim() seems to handle them gracefully with just a warning after proceeding.

Apologies that I do not have a handy reprex. I don't know if I can even recreate a parquet file with embedded nuls using arrow if it won't let me read one in, and I can't share this file due to company restrictions.

Please let me know how I can be of any more help!

Issue Links

• is duplicated by: ARROW-10324 function read_parquet(*,as_data_frame=TRUE) fails when embedded nuls present.
• is related to: ARROW-11682 [R] Regression from 2.0.0 -> 3.0.0: Null character in string prevents dataset from loading
• is related to: ARROW-11478 [R] Consider ways to make arrow.skip_nul option more user-friendly
• links to: GitHub Pull Request #8365