[ARROW-3801] [Python] Pandas-Arrow roundtrip makes pd categorical index not writeable

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.10.0
• Fix Version/s: 0.14.0
• Component/s: C++, Python
• External issue URL: https://github.com/apache/arrow/issues/20202
• Assignee: Unassigned
• Reporter: Thomas Buhrmann
• Created: 15/Nov/18 12:33
• Updated: 11/Jan/23 07:29
• Resolved: 07/Jun/19 16:00

Description

Serializing and deserializing a pandas series with categorical dtype will make the categorical index non-writeable, which in turn trips up pandas when e.g. reordering the categories, raising "ValueError: buffer source array is read-only" :

import pandas as pd
import pyarrow as pa

df = pd.Series([1,2,3], dtype='category', name="c1").to_frame()
print("DType before:", repr(df.c1.dtype))
print("Writeable:", df.c1.cat.categories.values.flags.writeable)
ro = df.c1.cat.reorder_categories([3,2,1])
print("DType reordered:", repr(ro.dtype), "\n")

tbl = pa.Table.from_pandas(df)
df2 = tbl.to_pandas()
print("DType after:", repr(df2.c1.dtype))
print("Writeable:", df2.c1.cat.categories.values.flags.writeable)
ro = df2.c1.cat.reorder_categories([3,2,1])
print("DType reordered:", repr(ro.dtype), "\n")

 

Outputs:

 

DType before: CategoricalDtype(categories=[1, 2, 3], ordered=False)
Writeable: True
DType reordered: CategoricalDtype(categories=[3, 2, 1], ordered=False)
DType after: CategoricalDtype(categories=[1, 2, 3], ordered=False)
Writeable: False
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-365-85b439586c1a> in <module>
 12 print("DType after:", repr(df2.c1.dtype))
 13 print("Writeable:", df2.c1.cat.categories.values.flags.writeable)
---> 14 ro = df2.c1.cat.reorder_categories([3,2,1])
 15 print("DType reordered:", repr(ro.dtype), "\n")