[ARROW-6313] [Format] Tracking for ensuring flatbuffer serialized values are aligned in stream/files.

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.15.0
• Component/s: Format
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/22693
• Assignee: Wes McKinney
• Reporter: Micah Kornfield
• Created: 22/Aug/19 03:33
• Updated: 11/Jan/23 07:46
• Resolved: 13/Sep/19 17:49

Description

Overall tracking bug for implementation for IPC/File format proposed by: https://github.com/apache/arrow/pull/4951/files

 

Implementations must support backwards compatibility with the old format (and ideally do memcopies when required to avoid undefined behavior).  Having a backwards compatible write mode is optional

Issue Links

• incorporates: ARROW-9848 [Rust] Implement changes to ensure flatbuffer alignment
• is a parent of: ARROW-6541 [Format][C++] Use two-part EOS and amend Format documentation
• is a parent of: ARROW-6545 [Go] Update Go IPC writer to use two-part EOS per mailing list discussion
• links to: GitHub Pull Request #5420

Sub-Tasks

• [C++] Implement changes to ensure flatbuffer alignment.: Resolved
• [Java] Make change to ensure flatbuffer reads are aligned: Resolved
• [Go] Make change to ensure flatbuffer reads are aligned: Resolved
• [JS] Implement changes to ensure flatbuffer alignment: Resolved
• [Integration] Update integration test to use generated binaries to ensure backwards compatibility: Resolved
• [Python] Provide mechanism for python to write out old format: Resolved
• [Java] Use IPC continuation token to mark EOS: Resolved
• [R] Provide mechanism to write out old format: Resolved