Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-6779

[Python] Conversion from datetime.datetime to timstamp('ns') can overflow

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Python
    • Labels:
      None

      Description

      In the python conversion of datetime scalars, there is no check for integer overflow:

      In [32]: pa.array([datetime.datetime(3000, 1, 1)], pa.timestamp('ns'))                                                                                                                                             
      Out[32]: 
      <pyarrow.lib.TimestampArray object at 0x7f5bddec00a8>
      [
        1830-11-23 00:50:52.580896768
      ]
      

      So in case the target type has nanosecond unit, this can give wrong results (I don't think the other resolutions can reach overflow, given the limited range of years of datetime.datetime).

      We should probably check for this case and raise an error.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jorisvandenbossche Joris Van den Bossche
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: