[ARROW-9777] [Rust] Implement IPC changes to catch up to 1.0.0 format

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 4.0.0
• Component/s: Rust
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25826
• Assignee: Neville Dipale
• Reporter: Neville Dipale
• Created: 17/Aug/20 23:53
• Updated: 11/Jan/23 08:08
• Resolved: 29/Aug/22 14:14

Description

There are a number of IPC changes and features which the Rust implementation has fallen behind on. It's effectively using the legacy format that was released in 0.14.x.

Some that I encountered are:

• change padding from 4 bytes to 8 bytes (along with the padding algorithm)
• add an IPC writer option to support the legacy format and updated format
• add error handling for the different metadata versions, we should support v4+ so it's an oversight to not explicitly return errors if unsupported versions are read

Some of the work already has Jiras open (e.g. body compression), I'll find them and mark them as related to this.

I'm tight for spare time, but I'll try work on this before the next release (along with the Parquet writer)

Issue Links

• incorporates: ARROW-9313 [Rust] Use feature enum
• is related to: ARROW-8676 [Rust] Create implementation of IPC RecordBatch body buffer compression from ARROW-300
• links to: GitHub Pull Request #9011
• links to: GitHub Pull Request #9126

Sub-Tasks

• [Rust] Update checked-in flatbuffer files: Resolved
• [Rust] Implement changes to ensure flatbuffer alignment: Resolved
• [Rust] Allow configuring flight IPC with IpcWriteOptions: Resolved
• [Rust] Support reading and writing V5 of IPC metadata: Resolved