[ARROW-5908] [C#] ArrowStreamWriter doesn't align buffers to 8 bytes

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.14.1, 0.15.0
• Component/s: C#
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/22320
• Assignee: Eric Erhardt
• Reporter: Eric Erhardt
• Created: 11/Jul/19 03:07
• Updated: 11/Jan/23 07:43
• Resolved: 11/Jul/19 22:06

Description

When writing RecordBatches using ArrowStreamWriter, if the ArrowBuffers being written aren't all 8 byte aligned, the serialized RecordBatch won't conform to the Arrow specification. This leads to other languages' readers to throw an error when reading Arrow streams written by the C# writer.

For example, if reading the stream from Python or C++, an error is raised here: 

https://github.com/apache/arrow/blob/f77c3427ca801597b572fb197b92b0133269049b/cpp/src/arrow/ipc/reader.cc#L107-L110

A similar error is raised when Java tries to read the stream.

We should be ensuring that the buffers being written to the stream are padded to 8 bytes, no matter their length, as specified in https://arrow.apache.org/docs/format/Layout.html#requirements-goals-and-non-goals

 

• It is required to have all the contiguous memory buffers in an IPC payload aligned at 8-byte boundaries. In other words, each buffer must start at an aligned 8-byte offset. Additionally, each buffer should be padded to a multiple of 8 bytes.

Issue Links

• links to: GitHub Pull Request #4851