[ARROW-7214] [Python] unpickling a pyarrow table with dictionary fields crashes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0, 0.14.1, 0.15.0, 0.15.1
• Fix Version/s: 0.16.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/23510
• Assignee: Joris Van den Bossche
• Reporter: Yevgeni Litvin
• Created: 20/Nov/19 07:53
• Updated: 11/Jan/23 07:52
• Resolved: 20/Nov/19 13:17

Description

The following code crashes on this check:

F1120 07:51:37.523720 12432 array.cc:773]  Check failed: (data->dictionary) != (nullptr) 

 

import cPickle as pickle
import pandas as pd
import pyarrow as pa

df = pd.DataFrame([{"cat": "a", "val":1},{"cat": "b", "val":2} ])
df["cat"] = df["cat"].astype('category')index_table = pa.Table.from_pandas(df, preserve_index=False)

with open('/tmp/zz.pickle', 'wb') as f:
    pickle.dump(index_table, f, protocol=2)

with open('/tmp/zz.pickle', 'rb') as f:
   index_table = pickle.load(f)

 

Used Python2 with the following environment:

Package         Version
--------------- -------
enum34          1.1.6  
futures         3.3.0  
numpy           1.16.5 
pandas          0.24.2 
pip             19.3.1 
pyarrow         0.14.1 (0.14.0 and up suffer from this issue)
python-dateutil 2.8.1  
pytz            2019.3 
setuptools      41.6.0 
six             1.13.0 

Issue Links

• links to: GitHub Pull Request #5871