[ARROW-9768] [Python] Pyarrow allows for unsafe conversions of datetime objects to timestamp nanoseconds

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.17.1, 1.0.0
• Fix Version/s: 2.0.0
• Component/s: Python
• Labels: pull-request-available
• Environment: OS: MacOSX Catalina ; Python Version: 3.7
• External issue URL: https://github.com/apache/arrow/issues/25817
• Assignee: Joris Van den Bossche
• Reporter: Joshua Lay
• Created: 17/Aug/20 05:08
• Updated: 11/Jan/23 08:08
• Resolved: 20/Aug/20 15:49

Description

Hi, 

In parquet, I want to store date values as timestamp format with nanoseconds precision. This works fine with most dates except those past pandas.Timestamp.max: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.max.html.

I was expecting some exception to be raised (like in Pandas), however this did not happen and the value was processed incorrectly. Note that this is with safe=True. Can this please be looked into? Thanks

Example Code:

pa.array([datetime(2262,4,12)], type=pa.timestamp("ns"))
{{}}

Return:

[
{{ 1677-09-21 00:25:26.290448384}}
]

Issue Links

• supercedes: ARROW-6779 [Python] Conversion from datetime.datetime to timstamp('ns') can overflow
• links to: GitHub Pull Request #7980
• links to: GitHub Pull Request #7987