[ARROW-4503] [C#] ArrowStreamReader allocates and copies data excessively

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Component/s: C#
• Labels: performance pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/21055
• Assignee: Eric Erhardt
• Reporter: Eric Erhardt
• Created: 08/Feb/19 00:28
• Updated: 11/Jan/23 07:34
• Resolved: 07/May/19 12:02

Description

When reading `RecordBatch` instances using the `ArrowStreamReader` class, it is currently allocating and copying memory 3 times for the data.

1. It is allocating memory in order to read the data from the Stream, and then reading from the Stream.  (This should be the only allocation that is necessary.)
2. It then creates a new `ArrowBuffer.Builder`, which allocates another `byte[]`, and calls `Append` on it, which copies the values to the new `byte[]`.
3. Finally, it then calls `.Build()` on the `ArrowBuffer.Builder`, which allocates memory from the MemoryPool, and then copies the intermediate buffer into it.

 

We should reduce this overhead to only allocating a single time (from the MemoryPool), and not copying the data more times than necessary.

Issue Links

• links to: GitHub Pull Request #3925