Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-6305

[Python] scalar pd.NaT incorrectly parsed in conversion from Python

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Python
    • Labels:
      None

      Description

      When converting from scalar values, using pd.NaT (the missing value indicator that pandas uses for datetime64 data) results in an incorrect timestamp:

      In [6]: pa.array([pd.Timestamp("2012-01-01"), pd.NaT]) 
      Out[6]: 
      <pyarrow.lib.TimestampArray object at 0x7f46c8368780>
      [
        2012-01-01 00:00:00.000000,
        0001-01-01 00:00:00.000000
      ]
      

      where pd.NaT is converted to "0001-01-01", which is strange, as that does not even correspond with the integer value of pd.NaT.

      Numpy's version (np.datetime64('NaT')) is correctly handled. Which also means that a pandas Series holding pd.NaT is handled correctly (as when converting to numpy it is using numpy's NaT).

      Related to ARROW-842.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jorisvandenbossche Joris Van den Bossche
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: