Details
-
New Feature
-
Status: Closed
-
Major
-
Resolution: Later
-
None
-
None
Description
For Arrow streams with unknown size, it would be useful to be able to write the data to disk either as a stream or as the file format (for random access) with minimal overhead; i.e. we would avoid record batch IPC loading and write the raw messages directly to disk