[ARROW-15142] Cannot mix struct and non-struct, non-null values error when saving nested types with PyArrow

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 6.0.1
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/30648

Language

Python

• Assignee: Unassigned
• Reporter: Karthik
• Created: 16/Dec/21 22:33
• Updated: 11/Jan/23 08:44

Description

When trying to save a Pandas dataframe with a nested type (list within list, list within dict) using pyarrow engine, the following error is encountered

ArrowInvalid: ('cannot mix list and non-list, non-null values', 'Conversion failed for column A with type object')

 

Repro:

import pandas as pd
x = pd.DataFrame({"A": [[24, 27, [1, 1]]]})
x.to_parquet('/tmp/a.pqt', engine="pyarrow")  

Doing a bit of googling, it appears that this is a known Arrow shortcoming. However, this is a commonly encountered datastructure, and 'fastparquet' handles this seamlessly. Is there a proposed timeline/plan for fixing this?