[ARROW-1743] [Python] Table to_pandas fails when index contains categorical column

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.1
• Fix Version/s: 0.8.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/17747
• Assignee: Licht Takeuchi
• Reporter: Brian Pendleton
• Created: 27/Oct/17 20:34
• Updated: 11/Jan/23 07:16
• Resolved: 14/Nov/17 03:47

Description

Categorical columns in the index of a dataframe are causing a roundtrip failure.

>>> df = pd.DataFrame({'a': [1, 2, 3], 'b': [1, 2, 3]})
>>> df['a'] = df.a.astype('category')
>>> df = df.set_index('a')
>>> tbl = pa.Table.from_pandas(df)
>>> tbl.to_pandas()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "table.pxi", line 881, in pyarrow.lib.Table.to_pandas
  File "C:\Users\bpendlet\Miniconda3\envs\panpy3\lib\site-packages\pyarrow\pandas_compat.py", line 303, in table_to_blockmanager
    if not values.flags.writeable:
AttributeError: 'Categorical' object has no attribute 'flags'

Works as expected when you don't change have the categorical:

>>> df = pd.DataFrame({'a': [1, 2, 3], 'b': [1, 2, 3]})
>>> df = df.set_index('a')
>>> tbl = pa.Table.from_pandas(df)
>>> tbl.to_pandas()
   b
a
1  1
2  2
3  3

Issue Links

• links to: GitHub Pull Request #1260