[ARROW-4512] [R] Stream reader/writer API that takes socket stream

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.12.0, 0.14.1, 1.0.0
• Fix Version/s: 8.0.0
• Component/s: R
• External issue URL: https://github.com/apache/arrow/issues/21063
• Assignee: Dewey Dunnington
• Reporter: Hyukjin Kwon
• Created: 08/Feb/19 07:29
• Updated: 11/Jan/23 07:34
• Resolved: 02/Jul/22 18:45

Description

I have been working on Spark integration with Arrow.

I realised that there are no ways to use socket as input to use Arrow stream format. For instance,
I want to something like:

connStream <- socketConnection(port = 9999, blocking = TRUE, open = "wb")

rdf_slices <- # a list of data frames.

stream_writer <- NULL
tryCatch({
  for (rdf_slice in rdf_slices) {
    batch <- record_batch(rdf_slice)
    if (is.null(stream_writer)) {
      stream_writer <- RecordBatchStreamWriter(connStream, batch$schema)  # Here, looks there's no way to use socket.
    }

    stream_writer$write_batch(batch)
  }
},
finally = {
  if (!is.null(stream_writer)) {
    stream_writer$close()
  }
})

Likewise, I cannot find a way to iterate the stream batch by batch

RecordBatchStreamReader(connStream)$batches()  # Here, looks there's no way to use socket.

This looks easily possible in Python side but looks missing in R APIs.

Issue Links

• duplicates: ARROW-9235 [R] Support for `connection` class when reading and writing files