[ARROW-5889] [Python][C++] Parquet backwards compat for timestamps without timezone broken

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 0.14.1, 0.15.0
• Component/s: C++, Python
• Labels: parquet pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/22303
• Assignee: TP Boudreau
• Reporter: Florian Jetter
• Created: 09/Jul/19 15:21
• Updated: 11/Jan/23 07:43
• Resolved: 13/Jul/19 17:55

Description

When reading a parquet file which has timestamp fields they are read as a timestamp with timezone UTC if the parquet file was written by pyarrow 0.13.0 and/or 0.12.1.

Expected behavior would be that they are loaded as timestamps without any timezone information.

The attached files contain one row for all basic types and a few nested types, the timestamp fields are called datetime64 and datetime64_tz

see also https://github.com/JDASoftwareGroup/kartothek/tree/master/reference-data/arrow-compat

https://github.com/JDASoftwareGroup/kartothek/blob/c47e52116e2dc726a74d7d6b97922a0252722ed0/tests/serialization/test_arrow_compat.py#L31

 

Issue Links

• links to: GitHub Pull Request #4856