[ARROW-5480] [Python] Pandas categorical type doesn't survive a round-trip through parquet

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.11.1, 0.13.0
• Fix Version/s: 0.15.0
• Component/s: Python
• Labels: pull-request-available
• Environment: python: 3.7.3.final.0 ; python-bits: 64 ; OS: Linux ; OS-release: 5.0.0-15-generic ; machine: x86_64 ; processor: x86_64 ; byteorder: little ; pandas: 0.24.2 ; numpy: 1.16.4 ; pyarrow: 0.13.0
• External issue URL: https://github.com/pandas-dev/pandas/issues/26616
• Assignee: Wes McKinney
• Reporter: Karl Dunkle Werner
• Created: 02/Jun/19 16:58
• Updated: 11/Jan/23 07:40
• Resolved: 19/Aug/19 18:13

Description

Writing a string categorical variable to from pandas parquet is read back as string (object dtype). I expected it to be read as category.
The same thing happens if the category is numeric – a numeric category is read back as int64.

In the code below, I tried out an in-memory arrow Table, which successfully translates categories back to pandas. However, when I write to a parquet file, it's not.

In the scheme of things, this isn't a big deal, but it's a small surprise.

import pandas as pd
import pyarrow as pa


df = pd.DataFrame({'x': pd.Categorical(['a', 'a', 'b', 'b'])})
df.dtypes  # category

# This works:
pa.Table.from_pandas(df).to_pandas().dtypes  # category

df.to_parquet("categories.parquet")
# This reads back object, but I expected category
pd.read_parquet("categories.parquet").dtypes  # object


# Numeric categories have the same issue:
df_num = pd.DataFrame({'x': pd.Categorical([1, 1, 2, 2])})
df_num.dtypes # category

pa.Table.from_pandas(df_num).to_pandas().dtypes  # category

df_num.to_parquet("categories_num.parquet")
# This reads back int64, but I expected category
pd.read_parquet("categories_num.parquet").dtypes  # int64

Issue Links

• is related to: ARROW-3652 [Python] CategoricalIndex is lost after reading back
• relates to: ARROW-3246 [Python][Parquet] direct reading/writing of pandas categoricals in parquet
• links to: GitHub Pull Request #5110