Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
In Python there is support for extension types that consists of a registration step that defines functions to handle metadata serialization and deserialization. In R, any extension name or metadata at the top level is currently obliterated on import. To implement geometry reading and writing to Parquet, IPC, and/or Feather, we will need to at the very least have the extension name and metadata preserved (in R), and at best provide a registration step to customize the behaviour of the resulting Array/DataType.
Reprex for R:
# remotes::install_github("paleolimbot/narrow") library(narrow) carray <- as_narrow_array(1:5) carray$schema$metadata[["ARROW:extension:name"]] <- "extension name!" carray$schema$metadata[["ARROW:extension:metadata"]] <- "bananas" carray$schema$metadata[["something else"]] <- "more bananas" array <- from_narrow_array(carray, arrow::Array) carray2 <- as_narrow_array(array) carray2$schema$metadata[["ARROW:extension:name"]] #> NULL carray2$schema$metadata[["ARROW:extension:metadata"]] #> NULL carray2$schema$metadata[["something else"]] #> NULL
There is some discussion of that as a solution to ARROW-14378, including an example of how pandas implements the 'interval' extension type (example contributed by jorisvandenbossche).
For the Interval example, there are some different parts living in different places:
- The Arrow Extension Type definition for pandas' interval type: https://github.com/pandas-dev/pandas/blob/fc6b441ba527ca32b460ae4f4f5a6802335497f9/pandas/core/arrays/_arrow_utils.py#L88-L136
- The _from_arrow_ implementation (doing the conversion to arrow): https://github.com/pandas-dev/pandas/blob/fc6b441ba527ca32b460ae4f4f5a6802335497f9/pandas/core/arrays/interval.py#L1405-L1455
- The _from_arrow_ implementation (conversion arrow -> pandas): https://github.com/pandas-dev/pandas/blob/fc6b441ba527ca32b460ae4f4f5a6802335497f9/pandas/core/dtypes/dtypes.py#L1227-L1255
Attachments
Issue Links
- links to
