[ARROW-9235] [R] Support for `connection` class when reading and writing files

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.0.0
• Component/s: R
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25332
• Assignee: Dewey Dunnington
• Reporter: Michael Quinn
• Created: 26/Jun/20 23:05
• Updated: 11/Jan/23 08:05
• Resolved: 22/Apr/22 16:59

Description

We have an internal filesystem that we interact with through objects that inherit from the connection class. These files aren't necessarily local, making it slightly more complicated to read and write parquet files, for example.

For now, we're generating raw vectors and using that to create the file. For example, to read files

ReadParquet <- function(filename, ...) {}}
   file <-file(filename,"rb")
   on.exit(close(file))
   raw <- readBin(file, "raw", FileInfo(filename)$size)
   return(arrow::read_parquet(raw, ...))
}

And to write,

WriteParquet <- function(df, filepath, ...) {
   stream <- BufferOutputStream$create()
   write_parquet(df, stream, ...)
   raw <- stream$finish()$data()
   file <- file(filepath, "wb")
   on.exit(close(file)
   writeBin(raw, file)
   return(invisible())
}

At the C++ level, we are interacting with ` R_new_custom_connection` defined here:
https://github.com/wch/r-source/blob/trunk/src/include/R_ext/Connections.h

I've been very impressed with how feature-rich arrow is. It would be nice to see this API supported as well.

Issue Links

• is duplicated by: ARROW-4512 [R] Stream reader/writer API that takes socket stream
• links to: GitHub Pull Request #12323