[ARROW-5125] [Python] Cannot roundtrip extreme dates through pyarrow

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.13.0
• Fix Version/s: 0.15.0
• Component/s: Python
• Labels: pull-request-available windows
• Environment: Windows 10, Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05)
• External issue URL: https://github.com/apache/arrow/issues/16705
• Assignee: Micah Kornfield
• Reporter: Max Bolingbroke
• Created: 05/Apr/19 07:36
• Updated: 11/Jan/23 07:38
• Resolved: 07/Sep/19 23:13

Description

You can roundtrip many dates through a pyarrow array:

 

>>> pa.array([datetime.date(1980, 1, 1)], type=pa.date32())[0]
datetime.date(1980, 1, 1)

 

But (on Windows at least), not extreme ones:

 

>>> pa.array([datetime.date(1960, 1, 1)], type=pa.date32())[0]
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "pyarrow\scalar.pxi", line 74, in pyarrow.lib.ArrayValue.__repr__
 File "pyarrow\scalar.pxi", line 226, in pyarrow.lib.Date32Value.as_py
OSError: [Errno 22] Invalid argument
>>> pa.array([datetime.date(3200, 1, 1)], type=pa.date32())[0]
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "pyarrow\scalar.pxi", line 74, in pyarrow.lib.ArrayValue.__repr__
 File "pyarrow\scalar.pxi", line 226, in pyarrow.lib.Date32Value.as_py

This is because datetime.utcfromtimestamp and datetime.timestamp fail on these dates, but it seems we should be able to totally avoid invoking this function when deserializing dates. Ideally we would be able to roundtrip these as datetimes too, of course, but it's less clear that this will be easy. For some context on this see https://bugs.python.org/issue29097.

This may be related to ARROW-3176 and ARROW-4746

Issue Links

• is related to: ARROW-4746 [C++/Python] PyDataTime_Date wrongly casted to PyDataTime_DateTime
• links to: GitHub Pull Request #5306