[ARROW-16838] [Python] Schema inference for pandas extension dtypes fails on indexes

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 8.0.0
• Fix Version/s: 10.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/32167
• Assignee: James Bourbeau
• Reporter: Ian Rose
• Created: 15/Jun/22 17:03
• Updated: 11/Jan/23 11:46
• Resolved: 21/Sep/22 07:52

Description

Hi! pa.Schema.from_pandas called on a dataframe whose index is a pandas extension dtype (e.g., string[python]) results in an error:

import pyarrow as pa
df = pd.DataFrame({"a": [1, 2]}, index=pd.Index(["A", "B"], dtype="string"))
pa.Schema.from_pandas(df)

produces

AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_1827952/3691394220.py in <module>
      1 import pyarrow as pa
      2 df = pd.DataFrame({"a": [1, 2]}, index=pd.Index(["A", "B"], dtype="string"))
----> 3 pa.Schema.from_pandas(df)

~/miniconda3/envs/dask/lib/python3.8/site-packages/pyarrow/types.pxi in pyarrow.lib.Schema.from_pandas()

~/miniconda3/envs/dask/lib/python3.8/site-packages/pyarrow/pandas_compat.py in dataframe_to_types(df, preserve_index, columns)
    527             type_ = pa.array(c, from_pandas=True).type
    528         elif _pandas_api.is_extension_array_dtype(values):
--> 529             type_ = pa.array(c.head(0), from_pandas=True).type
    530         else:
    531             values, type_ = get_datetimetz_type(values, c.dtype, None)

AttributeError: 'Index' object has no attribute 'head'

If I remove the `head` call, or convert the index to a series manually, things work.

Reported downstream in https://github.com/dask/dask/issues/9186

Related issue from a couple of years ago: https://issues.apache.org/jira/browse/ARROW-8159
 

Issue Links

• links to: GitHub Pull Request #14080