[ARROW-6305] [Python] scalar pd.NaT incorrectly parsed in conversion from Python

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/22686
• Assignee: Unassigned
• Reporter: Joris Van den Bossche
• Created: 21/Aug/19 08:48
• Updated: 11/Jan/23 07:46
• Resolved: 21/Aug/19 15:18

Description

When converting from scalar values, using pd.NaT (the missing value indicator that pandas uses for datetime64 data) results in an incorrect timestamp:

In [6]: pa.array([pd.Timestamp("2012-01-01"), pd.NaT]) 
Out[6]: 
<pyarrow.lib.TimestampArray object at 0x7f46c8368780>
[
  2012-01-01 00:00:00.000000,
  0001-01-01 00:00:00.000000
]

where pd.NaT is converted to "0001-01-01", which is strange, as that does not even correspond with the integer value of pd.NaT.

Numpy's version (np.datetime64('NaT')) is correctly handled. Which also means that a pandas Series holding pd.NaT is handled correctly (as when converting to numpy it is using numpy's NaT).

Related to ARROW-842.

Issue Links

• is duplicated by: ARROW-842 [Python] Handle more kinds of null sentinel objects from pandas 0.x