[ARROW-11480] [Python] Segmentation fault reading parquet with date filter with INT96 column

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 4.0.0
• Component/s: C++
• Labels: dataset pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/27362
• Assignee: Ben Kietzman
• Reporter: Henrik Anker Rasmussen
• Created: 03/Feb/21 10:36
• Updated: 11/Jan/23 08:20
• Resolved: 16/Feb/21 14:59

Description

If I read a parquet file (see attachment) with timestamps generated in Spark and apply a filter on a date column I get segmentation fault
 

import pyarrow.parquet as pq  
now = datetime.datetime.now()
table = pq.read_table("timestamp.parquet", filters=[("date", "<=", now)])

 

The attached parquet file is generated with this code in spark:

now = datetime.datetime.now() 
data = {"date": [ now - datetime.timedelta(days=i) for i in range(100)]} 
schema = { "type": "struct", "fields": [{"name": "date", "type": "timestamp", "nullable": True, "metadata": {}}, ], } 
spf = spark.createDataFrame(pd.DataFrame(data), schema=StructType.fromJson(schema)) 
spf.write.format("parquet").mode("overwrite").save("timestamp.parquet") 

If I downgrade pyarrow to 2.0.0 it works fine.

Python version 3.7.7

pyarrow version 3.0.0

Issue Links

• duplicates: ARROW-11538 [Python] Segfault reading Parquet dataset with Timestamp filter
• links to: GitHub Pull Request #9470