[ARROW-4492] [Python] Failure reading Parquet column as pandas Categorical in 0.12

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.12.0
• Fix Version/s: 0.12.1
• Component/s: Python
• Labels: Parquet
• External issue URL: https://github.com/apache/arrow/issues/21046
• Assignee: Unassigned
• Reporter: George Sakkis
• Created: 06/Feb/19 08:37
• Updated: 11/Jan/23 07:34
• Resolved: 06/May/19 10:20

Description

On pyarrow 0.12.0 some (but not all) columns cannot be read as category dtype. Attached is an extracted failing sample.

 

import dask.dataframe as dd
df = dd.read_parquet('slug.pq', categories=['slug'], engine='pyarrow').compute()
print(len(df['slug'].dtype.categories))
 

This works on pyarrow 0.11.1 (and fastparquet 0.2.1).

Issue Links

• is blocked by: ARROW-4872 [Python] Keep backward compatibility for ParquetDatasetPiece