Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-2660

[Python] Experiment with zero-copy pickling

    XMLWordPrintableJSON

    Details

      Description

      PEP 574 has an implementation ready and a PyPI-available backport (at https://pypi.org/project/pickle5/ ). Adding experimental support for it would allow for zero-copy pickling of Arrow arrays, columns, etc.

      I think it mainly involves implementing reduce_ex on the Buffer class, as described in https://www.python.org/dev/peps/pep-0574/#producer-api

      In addition, the consumer API added by PEP 574 could be used in Arrow's serialization layer, to avoid or minimize copies when serializing foreign objects.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                pitrou Antoine Pitrou
                Reporter:
                pitrou Antoine Pitrou
              • Votes:
                1 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3h 10m
                  3h 10m