[ARROW-2711] [Python/C++] Pandas-Arrow doesn't roundtrip when column of lists has empty first element

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.10.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/19101
• Assignee: Antoine Pitrou
• Reporter: Thomas Buhrmann
• Created: 14/Jun/18 17:46
• Updated: 11/Jan/23 07:22
• Resolved: 23/Jul/18 16:53

Description

Hi, I thought this had been fixed in the past, but this simple use case still breaks:

 

df = pd.DataFrame(dict(x=[[], ["a"]]))
tbl = pyarrow.Table.from_pandas(df)
print(tbl.schema)

results in a wrong inferred type of "list<item: null>":

 

x: list<item: null>
  child 0, item: null
__index_level_0__: int64
metadata
--------
{b'pandas': b'{"index_columns": ["__index_level_0__"], "column_indexes": [{"na'
            b'me": null, "field_name": null, "pandas_type": "unicode", "numpy_'
            b'type": "object", "metadata": {"encoding": "UTF-8"}}], "columns":'
            b' [{"name": "x", "field_name": "x", "pandas_type": "list[empty]",'
            b' "numpy_type": "object", "metadata": null}, {"name": null, "fiel'
            b'd_name": "__index_level_0__", "pandas_type": "int64", "numpy_typ'
            b'e": "int64", "metadata": null}], "pandas_version": "0.22.0"}'}

When converting the Table back to pandas all elements are now None too:

 

df2 = tbl.to_pandas()
print(df2)

       x

0     [] 
1 [None]

 

 

Issue Links

• links to: GitHub Pull Request #2309