Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-3587

[Python] Efficient serialization for Arrow Objects (array, table, tensor, etc)

    XMLWordPrintableJSON

Details

    Description

      Currently, Arrow seems to have poor serialization support for its own objects.

      For example,
       

      import pyarrow 
      arr = pyarrow.array([1, 2, 3, 4]) 
      pyarrow.serialize(arr)
      

      Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "pyarrow/serialization.pxi", line 337, in pyarrow.lib.serialize
      File "pyarrow/serialization.pxi", line 136, in pyarrow.lib.SerializationContext._serialize_callback
      pyarrow.lib.SerializationCallbackError: pyarrow does not know how to serialize objects of type <class 'pyarrow.lib.Int64Array'>.

      I am working Ray & modin project, using plasma to store Arrow objects. Lack of direct serialization support harms the performance, so I would like to push a PR to fix this problem.
      I wonder if it is welcome or is there someone else doing it?

      Attachments

        Issue Links

          Activity

            People

              suquark Siyuan Zhuang
              suquark Siyuan Zhuang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 5.5h
                  5.5h