[ARROW-17058] Timezone aware parquet read with schema and filters

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 8.0.0
• Component/s: Parquet, Python
• External issue URL: https://github.com/apache/arrow/issues/32366

Language

Python

• Assignee: Unassigned
• Reporter: Blaž Zupančič
• Created: 12/Jul/22 18:05
• Updated: 11/Jan/23 11:48

Description

The parquet.read_table() method in pyarrow 8.0.0 added `schema` parameter which is great for handling timestamps, i.e., they are correctly converted from UTC to the timezone specified in the schema.

However, when `schema` is used together with `filters`, timezone conversion fails with "Cannot compare timestamp with timezone to timestamp without timezone" error. This was tested on 2 files created with different versions of spark. The test code, files and the output are attached.