[ARROW-474] Create an Arrow streaming file fomat

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.2.0
• Component/s: Format
• External issue URL: https://github.com/apache/arrow/issues/16118
• Assignee: Nong Li
• Reporter: Nong Li
• Created: 11/Jan/17 01:02
• Updated: 11/Jan/23 07:09
• Resolved: 19/Jan/17 19:46

Description

It would be helpful to have a file/container layout that allows for streaming consumer and producers of arrow data.

As a motivating example, a reader could have this API:
Iterator<Record> read(java.io.InputStream source).
Similar a writer can have:
void write(RecordBatch, java.io.OutputStream dest)

The streams can be from a variety of sources: it be files, over the network, shared memory, pipe, etc. The layout would make it possible to implement
both APIs without requiring intermediate buffering.

Speaking with Julien, this involves creating a simple container format similar to ARROW-264 but suitable for streaming consumer and producers. The layout in ARROW-264 has the schema at the end so can't be used in a streaming way. This format would likely be a rearranging of the current File layout pieces, something similar to:

MAGIC, schema, totalBatches, (RecordBatchHeader, RecordBatchBody)*, MAGIC (or some end marker)

totalBatches is optional for streaming producers that don't know.