[ARROW-7856] [Python] to_pandas() causing datetimes > pd.Timestamp.max to wrap around

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.15.1
• Component/s: Python

Environment

Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic

Python 3.7.3

In [3]: pa.__version__
Out[3]: '0.15.1'

In [4]: pd.__version__
Out[4]: '0.25.2'

• Flags: Important
• External issue URL: https://github.com/apache/arrow/issues/24084
• Assignee: Unassigned
• Reporter: Kevin Glasson
• Created: 14/Feb/20 06:02
• Updated: 11/Jan/23 07:56
• Resolved: 14/Feb/20 12:43

Description

When writing a dataframe containing `datetime.datetime` in an object columns any datetime that is greater than pd.Timestamp.max or less than pd.Timestamp.min is wrapped around.

 

For reference these are the timestamp min and max values.

 

In [43]: pd.Timestamp.max
Out[43]: Timestamp('2262-04-11 23:47:16.854775807')
In [44]: pd.Timestamp.min
Out[44]: Timestamp('1677-09-21 00:12:43.145225')

 

 

To reproduce the error using pandas

 

In [49]: df = pd.DataFrame({"A":[datetime.datetime(2262,4,12)]})

In [50]: df
Out[50]:
                     A
0  2262-04-12 00:00:00

In [51]: df.to_parquet("datetimething.parquet")

In [52]: pd.read_parquet("datetimething.parquet")
Out[52]:
                              A
0 1677-09-21 00:25:26.290448384

I have narrowed it down as far as to note that it is happening when converting a `pa.Table` using the `to_pandas()` method.

In [30]: df = pd.DataFrame({"A":[datetime.datetime(2262,4,12)]})
In [31]: tf = pa.Table.from_pandas(df)
In [32]: tf.columns
Out[32]: [<pyarrow.lib.ChunkedArray object at 0x7f23884deef8>
 [
   [
     2262-04-12 00:00:00.000000
   ]
 ]
]
In [33]: tf.to_pandas()
Out[33]:                      A
0 1677-09-21 00:25:26.290448384

Issue Links

• is related to: ARROW-5359 [Python] timestamp_as_object support for pa.Table.to_pandas in pyarrow
• is related to: ARROW-7758 [Python] Wrong conversion of timestamps that are out of bounds for pandas (eg 0000-01-01)