Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
If I open a dataset of CSVs with a schema, the error message tells me to supply `skip = 1` if my data contains a header row (to prevent it being read in as data), but only skip_rows = 1 actually works.
library(arrow) library(dplyr) td <- tempfile() dir.create(td) write_dataset(mtcars, td, format = "csv") schema <- schema(mpg = float64(), cyl = float64(), disp = float64(), hp = float64(), drat = float64(), wt = float64(), qsec = float64(), vs = float64(), am = float64(), gear = float64(), carb = float64()) open_dataset(td, format = "csv", schema = schema) %>% collect() #> Error in `handle_csv_read_error()`: #> ! Invalid: Could not open CSV input source '/tmp/RtmppZbpeF/file6cec135ed29c/part-0.csv': Invalid: In CSV column #0: Row #1: CSV conversion error to double: invalid value 'mpg' #> /home/nic2/arrow/cpp/src/arrow/csv/converter.cc:550 decoder_.Decode(data, size, quoted, &value) #> /home/nic2/arrow/cpp/src/arrow/csv/parser.h:123 status #> /home/nic2/arrow/cpp/src/arrow/csv/converter.cc:554 parser.VisitColumn(col_index, visit) #> /home/nic2/arrow/cpp/src/arrow/csv/reader.cc:463 arrow::internal::UnwrapOrRaise(maybe_decoded_arrays) #> /home/nic2/arrow/cpp/src/arrow/compute/exec/exec_plan.cc:445 iterator_.Next() #> /home/nic2/arrow/cpp/src/arrow/record_batch.cc:336 ReadNext(&batch) #> /home/nic2/arrow/cpp/src/arrow/record_batch.cc:347 ReadAll(&batches) #> ℹ If you have supplied a schema and your data contains a header row, you should supply the argument `skip = 1` to prevent the header being read in as data. open_dataset(td, format = "csv", schema = schema, skip = 1) %>% collect() #> Error: The following option is supported in "read_delim_arrow" functions but not yet supported here: "skip" open_dataset(td, format = "csv", schema = schema, skip_rows = 1) %>% collect() #> # A tibble: 32 × 11 #> mpg cyl disp hp drat wt qsec vs am gear carb #> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> #> 1 21 6 160 110 3.9 2.62 16.5 0 1 4 4 #> 2 21 6 160 110 3.9 2.88 17.0 0 1 4 4 #> 3 22.8 4 108 93 3.85 2.32 18.6 1 1 4 1 #> 4 21.4 6 258 110 3.08 3.22 19.4 1 0 3 1 #> 5 18.7 8 360 175 3.15 3.44 17.0 0 0 3 2 #> 6 18.1 6 225 105 2.76 3.46 20.2 1 0 3 1 #> 7 14.3 8 360 245 3.21 3.57 15.8 0 0 3 4 #> 8 24.4 4 147. 62 3.69 3.19 20 1 0 4 2 #> 9 22.8 4 141. 95 3.92 3.15 22.9 1 0 4 2 #> 10 19.2 6 168. 123 3.92 3.44 18.3 1 0 4 4 #> # … with 22 more rows
Attachments
Issue Links
- links to
