Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-10110

[Rust] Add support to consume C Data Interface

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • Rust

    Description

      The goal of this issue is to support consuming C Data arrays from Rust using FFI.

      The use-case that motivated this issue was the possibility of running DataFusion from Python and support moving arrays from DataFusion to Python/Pyarray and vice-versa.

      In particular, so that users can write Python UDFs that expect arrow arrays and return arrow arrays, in the same spirit as pandas-udfs in Spark work for Pandas.

      The brute-force way of writing these arrays is by converting element by element from and to native types. The efficient way of doing it to pass the memory address from and to each implementation, which is zero-copy.

      To support the latter, we need an FFI implementation in Rust that produces and consumes C's Data interface

      Attachments

        Issue Links

          Activity

            jorgecarleitao Jorge Leitão added a comment -

            Addressed by 1d2b4a5

            jorgecarleitao Jorge Leitão added a comment - Addressed by 1d2b4a5
            rokm Rok Mihevc added a comment -

            This issue has been migrated to issue #26124 on GitHub. Please see the migration documentation for further details.

            rokm Rok Mihevc added a comment - This issue has been migrated to issue #26124 on GitHub. Please see the migration documentation for further details.

            People

              jorgecarleitao Jorge Leitão
              jorgecarleitao Jorge Leitão
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 4.5h
                  4.5h