[ARROW-10511] [Python] Table.to_pandas() failing when timezone-awareness mismatch in metadata

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 3.0.0
• Component/s: Python
• Labels: pull-request-available
• Environment: Ubuntu 20.04, Python 3.8.6, Pandas 1.1.4
• External issue URL: https://github.com/apache/arrow/issues/26481
• Assignee: Joris Van den Bossche
• Reporter: Karl Dunkle Werner
• Created: 06/Nov/20 19:13
• Updated: 11/Jan/23 08:13
• Resolved: 16/Nov/20 15:04

Description

We're having an issue with timezones in the Table to_pandas methods. See example below.

import pyarrow as pa
import pandas as pd

print(pa.__version__)
# 2.0.0

df = pd.DataFrame({"time": pd.to_datetime([0, 0])})

time_field = pa.field("time",type=pa.timestamp("ms", tz="utc"), nullable=False)
schema = pa.schema([time_field])

tab = pa.Table.from_pandas(df, schema)

tab.to_pandas() 

# File ".../pandas_compat.py", line 777, in table_to_blockmanager
#   table = _add_any_metadata(table, pandas_metadata)
# File ".../pandas_compat.py", line 1184, in _add_any_metadata
#   tz = col_meta['metadata']['timezone']
# TypeError: 'NoneType' object is not subscriptable

Related issues:
https://github.com/catalyst-cooperative/pudl/issues/705

Issue Links

• links to: GitHub Pull Request #8625