[ARROW-13421] [C++] Add functionality for reading in columns as floats from delimited files where a comma has been used as a decimal separator

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 6.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/18752
• Assignee: Antoine Pitrou
• Reporter: Daniel Paierl
• Created: 21/Jul/21 08:39
• Updated: 11/Jan/23 08:33
• Resolved: 03/Aug/21 11:41

Description

When reading in data where commas have been used as decimal separators (e.g. 3,141 to indicate pi), the column is read in as a character string. If I try to specify a schema in R, i.e.:

# create table of values with decimal places
set.seed(1)
tbl <- tibble::tibble(x = rnorm(5))
# write to disk with comma separator
readr::write_csv2(tbl, "tst.csv")
# read back in
read_delim_arrow("tst.csv", delim = ";", schema = schema(x = float32()))

I get the following error:

Error: Invalid: In CSV column #0: CSV conversion error to float: invalid value 'x'
/home/nic2/arrow/cpp/src/arrow/csv/converter.cc:437 decoder_.Decode(data, size, quoted, &value)
/home/nic2/arrow/cpp/src/arrow/csv/parser.h:84 status
{{/home/nic2/arrow/cpp/src/arrow/csv/converter.cc:441 parser.VisitColumn(col_index, visit) }}

Please can we have the functionality to be able to read in data from this format as it's fairly common across a number of countries?

Issue Links

• links to: GitHub Pull Request #10852