[ARROW-2273] Cannot deserialize pandas SparseDataFrame

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.10.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/18230
• Assignee: Licht Takeuchi
• Reporter: Mitar
• Created: 06/Mar/18 12:41
• Updated: 11/Jan/23 07:20
• Resolved: 05/May/18 08:48

Description

>>> import pyarrow
>>> import pandas
>>> a = pandas.SparseDataFrame(

{'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]}

)
>>> pyarrow.deserialize(pyarrow.serialize(a).to_buffer())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "serialization.pxi", line 441, in pyarrow.lib.deserialize
File "serialization.pxi", line 404, in pyarrow.lib.deserialize_from
File "serialization.pxi", line 257, in pyarrow.lib.SerializedPyObject.deserialize
File "serialization.pxi", line 174, in pyarrow.lib.SerializationContext._deserialize_callback
File ".../.virtualenv/arrow/lib/python3.6/site-packages/pyarrow/serialization.py", line 77, in _deserialize_pandas_dataframe
return pdcompat.serialized_dict_to_dataframe(data)
File ".../.virtualenv/arrow/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 450, in serialized_dict_to_dataframe
for block in data['blocks']]
File ".../.virtualenv/arrow/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 450, in <listcomp>
for block in data['blocks']]
File ".../.virtualenv/arrow/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 478, in _reconstruct_block
block = _int.make_block(block_arr, placement=placement)
File ".../.virtualenv/arrow/lib/python3.6/site-packages/pandas/core/internals.py", line 2957, in make_block
return klass(values, ndim=ndim, fastpath=fastpath, placement=placement)
File ".../.virtualenv/arrow/lib/python3.6/site-packages/pandas/core/internals.py", line 120, in _init_
len(self.mgr_locs)))
ValueError: Wrong number of items passed 3, placement implies 1

Issue Links

• links to: GitHub Pull Request #1997