[ARROW-1654] [Python] pa.DataType cannot be pickled

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/17662
• Assignee: Wes McKinney
• Reporter: Li Jin
• Created: 05/Oct/17 23:40
• Updated: 11/Jan/23 07:16
• Resolved: 23/Oct/17 22:18

Description

In [26]: t
Out[26]: DataType(int64)

In [25]: pickle.dumps(t)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-25-f90063f6658b> in <module>()
----> 1 pickle.dumps(t)

/home/icexelloss/miniconda3/envs/spark-dev/lib/python3.5/site-packages/pyarrow/lib.cpython-35m-x86_64-linux-gnu.so in pyarrow.lib.DataType._reduce_cython_()

TypeError: no default _reduce_ due to non-trivial _cinit_

This is discovered when trying to send a pa.DataType along with a udf in pyspark. The workaround is to send pyspark DataType and convert to pa.DataType. It would be nice to able to pickle pa.DataType.

Issue Links

• links to: GitHub Pull Request #1238