[ARROW-16547] [Python] to_pandas fails with FixedOffset timezones when timestamp_as_object is used

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 11.0.0
• Component/s: Python
• Labels: pull-request-available python-conversion
• Flags: Important
• External issue URL: https://github.com/apache/arrow/issues/31908
• Assignee: Alenka Frim
• Reporter: Sander Goos
• Created: 12/May/22 15:15
• Updated: 11/Jan/23 11:44
• Resolved: 27/Oct/22 07:56

Description

The `to_pandas` method fails with "ValueError: fromutc: dt.tzinfo is not self" when timestamp_as_object=True and a timezone with a fixed offset is used. E.g. "+08:00"

Repro script attached.

 

The problem seems to be that `fromutc` is called on the tzinfo object here, which is not working when the object is pytz._FixedOffset: https://github.com/apache/arrow/blob/90aac16761b7dbf5fe931bc8837cad5116939270/cpp/src/arrow/python/arrow_to_pandas.cc#L1068

import pyarrow as pa
import datetime as dt
import pytz

tz = pytz.FixedOffset(120)
ts = tz.localize(dt.datetime(2022, 5, 12, 16, 57))

timestamps = pa.array([ts])
names = ["timestamp_col"]
table = pa.Table.from_arrays([timestamps], names=names)

print(table.schema)

# Works fine
print(table.to_pandas())

# Fails with "ValueError: fromutc: dt.tzinfo is not self"
table.to_pandas(timestamp_as_object=True)

Issue Links

• links to: GitHub Pull Request #14448