Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-11696

[Rust] Create record batches from in memory IPC without memory copies

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Invalid
    • None
    • None
    • Rust
    • None

    Description

      I have the IPC format of Arrow record batches in memory (e.g. as &[u8]) and would like to create a vector of batches while avoiding as many memory copies as possible. It would be great if there was a way to create the vector without having to go through the file abstraction.

      I might be misunderstanding the way the file reader works and maybe it does not incur memory copies. I think it does, though, since creating arrow record batches from a larger arrow buffer takes much longer.

      Attachments

        Activity

          People

            Unassigned Unassigned
            domoritz Dominik Moritz
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: