[ARROW-15737] pyarrow.parquet.read_table("parquet_file") causes bus error in ipython

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 7.0.0
• Component/s: Python
• Labels: parquet pyarrow
• Environment: macOS 12.2.1 aarch64 ; python. 3.10.1 ; arrow 7.0.0
• External issue URL: https://github.com/apache/arrow/issues/31185
• Assignee: Unassigned
• Reporter: Jay Edwards
• Created: 20/Feb/22 23:37
• Updated: 11/Jan/23 11:39

Description

I have a parquet file with two columns (int64 and double) and 9 million rows. The parquet tools (parquet, parquet-reader, parquet-schema...) read it perfectly. (I have many files, actually, but they all exhibit the same behavior).

The following code fails with "zsh bus error  ipython":

import pyarrow.parquet as pq
pq.read_table("parquet_file")

These snippets work properly.

pq.read_table("parquet_file", use_lagacy_dataset=True)

f = pq.ParquetFile("parquet_file")
f.read()
for batch in f.iterbatches():
print(len(batch))