[ARROW-10232] FixedSizeListArray is incorrectly written/read to/from parquet

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.0.1
• Fix Version/s: 2.0.0
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/18318
• Assignee: Unassigned
• Reporter: Simon Perkins
• Created: 08/Oct/20 10:11
• Updated: 11/Jan/23 08:11
• Resolved: 08/Oct/20 10:50

Description

FixedSizeListArray's seem to be either incorrectly written or read to or from Parquet files.

 

When reading the parquet file, nulls/Nones are returned where the original values should be.

 

import pyarrow as pa
import pyarrow.parquet as pq
import numpy as np

np_data = np.arange(20*4).reshape(20, 4).astype(np.float64)
pa_data = pa.FixedSizeListArray.from_arrays(np_data.ravel(), 4)
assert np_data.tolist() == pa_data.tolist()

schema = pa.schema([pa.field("rectangle", pa_data.type)])
table = pa.table({"rectangle": pa_data}, schema=schema)
pq.write_table(table, "test.parquet")

in_table = pq.read_table("test.parquet")   
# rectangle is filled with nulls
assert in_table.column("rectangle").to_pylist() == pa_data.tolist()

 

Issue Links

• is superceded by: ARROW-7960 [C++][Parquet] Add support for schema translation from parquet nodes back to arrow for missing types