[ARROW-264] Create an Arrow File format

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.1.0
• Component/s: Format
• External issue URL: https://github.com/apache/arrow/issues/15605
• Assignee: Julien Le Dem
• Reporter: Julien Le Dem
• Created: 19/Aug/16 20:50
• Updated: 11/Jan/23 07:07
• Resolved: 26/Aug/16 15:20

Description

File layout:
(DictionaryBatch, RecordBatch, Schema as defined in Message.fbs)

MAGIC:   ARROW1
(
DictionaryBatch:  DictionaryBatch Header (FlatBuffer)
DictionaryBatch: DictionaryBatch Body (buffers concatenated)
)*
(
RecordBacth: RecordBatch Header (FlatBuffer)
RecordBacth: RecordBatch Body (buffers concatenated)
)+
Footer: Flatbuffer
Footer length: int (4 bytes unsigned LE)
MAGIC: ARROW1

Footer definition:

table Footer {
  schema: org.apache.arrow.flatbuf.Schema;
  dictionaries: [ Block ];
  recordBatches: [ Block ];
}
struct Block {
  offset: long;
  metaDataLength: int;
  bodyLength: long;
}

Issue Links

• is depended upon by: ARROW-267 [C++] C++ implementation of file-like layout for RPC / IPC
• links to: PR