Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Duplicate
-
9.0.0
-
None
-
None
Description
Arrow 9.0 R client introduced support for dates with lubridate (and base R as.Date()) functions, which is awesome.
However, these functions fail to handle decimal dates. This will especially confuse R users because the native R functions work as expected, and R users will not realize the metaprogramming translation. Easiest to see this in a minimal reprex:
library(arrow); library(lubridate); library(dplyr)
f <- tempfile()
data.frame(t = Sys.time(), A = 1) |>
write_dataset(f, partitioning = "t")
# ERRORS
open_dataset(f) |> mutate(as_date(t)) |> collect()
This errors with message:
open_dataset(f) |> mutate(as_date(t)) |> collect()
Error in `collect()`:
! Invalid: Failed to parse string: '2022-09-30 22:03:32.123248' as a scalar of type timestamp[s]
Which is strange because lubridate::as_date('2022-09-30 22:03:32.123248') works fine.
It's easy to see the cause of the error prior to collect:
as_date(t): date32[day] (cast(strptime(t, {format="%Y-%m-%d", unit=SECOND, error_is_null=false}), {to_type=date32[day], allow_int_overflow=false, allow_time_truncate=false, allow_time_overflow=false, allow_decimal_truncate=false, allow_float_truncate=false, allow_invalid_utf8=false}))
We can see a lot of assumptions there about units of parsing, but afaik from R we have no way to control them. The issue is particularly ironic because as you see in my example, the column has only become a string because we used it as a partition. So arrow coerced the timestamp to a string originally (using microsecond precision – which is an understandable choice because it is loss-less, though it is different from R's as.character() behavior). But ironically, now arrow doesn't understand how to reverse it's own timestamp->string behavior to get a back to a timestamp!
Ideally the user would have more control of these, and the default assumptions would be consistent. Ideally, as_datetime, as_date, etc should not choke regardless of the precision of the seconds, matching the existing behavior of the base R (as.Date etc) and lubridate functions.