Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-4502

[C#] Add support for zero-copy reads

    XMLWordPrintableJSON

    Details

      Description

      In the Python (and C++) API, you can create a `RecordBatchStreamReader`, and if you give it an `InputStream` that supports zero-copy reads, you can get back `RecordBatch` objects without allocating new memory and copying all the data.

      There is currently no way to read Arrow RecordBatch instances without allocating new memory and copying all the data. We should enable this scenario in the C# API.

       

      My proposal is to create a new `class ArrowRecordBatchReader : IArrowReader`. It's constructor will take a `ReadOnlyMemory<byte> data` parameter, and it will be able to read `RecordBatch` instances just like the existing `ArrowStreamReader`. As part of this new class, we will refactor any common code out of `ArrowStreamReader` in order for the parsing logic to be shared, where necessary.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                eerhardt Eric Erhardt
                Reporter:
                eerhardt Eric Erhardt
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 72h
                  72h
                  Remaining:
                  Time Spent - 9h 20m Remaining Estimate - 23h
                  23h
                  Logged:
                  Time Spent - 9h 20m Remaining Estimate - 23h Time Not Required
                  9h 20m