[ARROW-18099] [Python] Cannot create pandas categorical from table only with nulls

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 9.0.0
• Component/s: Python
• Labels: pull-request-available python-conversion
• Environment: OSX 12.6 ; M1 silicon
• External issue URL: https://github.com/apache/arrow/issues/33297
• Assignee: Damian Barabonkov
• Reporter: Damian Barabonkov
• Created: 19/Oct/22 13:32
• Updated: 11/Jan/23 11:58

Description

A pyarrow Table with only null values cannot be instantiated as a Pandas DataFrame with said column as a category. However, pandas does support "empty" categoricals. Therefore, a simple patch would be to load the pa.Table as an object first and convert, once in pandas, to a categorical which will be empty. However, that does not solve the pyarrow bug at its root.

 

Sample reproducible example

import pyarrow as pa
pylist = [{'x': None, '__index_level_0__': 2}, {'x': None, '__index_level_0__': 3}]
tbl = pa.Table.from_pylist(pylist)
 

# Errors

df_broken = tbl.to_pandas(categories=["x"])
 

# Works
df_works = tbl.to_pandas()
df_works = df_works.astype({"x": "category"}) 

Issue Links

• links to: GitHub Pull Request #14953