[ARROW-7883] [Python] pyarrow.serialize does not support pandas nullable integer type

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: Python
• Labels: pyarrow-serialization
• External issue URL: https://github.com/apache/arrow/issues/24108
• Assignee: Unassigned
• Reporter: Benjamin
• Created: 19/Feb/20 09:57
• Updated: 11/Jan/23 07:56
• Resolved: 17/Feb/21 16:01

Description

Serializing an IntegerArray doesn't seem to work with the latest version of pandas and pyarrow

import pandas as pd
import pyarrow  # version 0.16
import pyarrow as pa

# workaround suggested in https://issues.apache.org/jira/browse/ARROW-5379
pd.arrays.IntegerArray.__arrow_array__ = lambda self, type: pyarrow.array(self._data, mask=self._mask, type=type)

df = pd.DataFrame([1, 2])
df = df.convert_dtypes()

# following https://arrow.apache.org/docs/python/ipc.html#serializing-pandas-objects
context = pa.default_serialization_context()
context.serialize(df) 

 SerializationCallbackError: pyarrow does not know how to serialize objects of type <class 'pandas.core.arrays.integer.IntegerArray'>

xref https://stackoverflow.com/q/60285486/2146052