[ARROW-8812] [Python] Column names of type CategoricalIndex fails to convert back to pandas

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 0.15.1
• Component/s: Python
• Labels: pandas parquet
• Environment: Python 3.7.7 ; MacOS (Darwin-19.4.0-x86_64-i386-64bit) ; Pandas 1.0.3 ; Pyarrow 0.15.1
• External issue URL: https://github.com/apache/arrow/issues/24955
• Assignee: Unassigned
• Reporter: Jonas Nelle
• Created: 15/May/20 08:49
• Updated: 11/Jan/23 08:02

Description

When columns are of type CategoricalIndex, saving and reading the table back causes a TypeError: data type "categorical" not understood:

import pandas as pd
from pyarrow import parquet, Table

base_df = pd.DataFrame([['foo', 'j', "1"],
                        ['bar', 'j', "1"],
                        ['foo', 'j', "1"],
                        ['foobar', 'j', "1"]],
                       columns=['my_cat', 'var', 'for_count'])

base_df['my_cat'] = base_df['my_cat'].astype('category')

df = (
    base_df
    .groupby(["my_cat", "var"], observed=True)
    .agg({"for_count": "count"})
    .rename(columns={"for_count": "my_cat_counts"})
    .unstack(level="my_cat", fill_value=0)
)

print(df)

The resulting data frame looks something like this:

	my_cat_counts
my_cat	foo	bar	foobar
var
j	2	1	1

Then, writing and reading causes the KeyError:

parquet.write_table(Table.from_pandas(df), "test.pqt")
parquet.read_table("test.pqt").to_pandas()
> TypeError: data type "categorical" not understood

In the example, the column is also a MultiIndex, but that isn't the problem:

df.columns = df.columns.get_level_values(1)
parquet.write_table(Table.from_pandas(df), "test.pqt")
parquet.read_table("test.pqt").to_pandas()
> TypeError: data type "categorical" not understood

This is the workaround suggested on stackoverflow:

df.columns = pd.Index(list(df.columns)) # suggested fix for the time being
parquet.write_table(Table.from_pandas(df), "test.pqt")
parquet.read_table("test.pqt").to_pandas() # no error

Are there any plans to support the pattern described here in the future?