Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-991

[Python] PyArray_SimpleNew should not be used with NPY_DATETIME

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.4.0
    • Component/s: Python
    • Labels:
      None

      Description

      From a non-deterministic test failure in pandas reported to me by Jeff Reback, I see that PyArray_SimpleNew calls PyArray_DescrFromType, which returns a reference to a cached dtype: https://github.com/numpy/numpy/blob/1d592c12ca7f9c7f471aa8d20b538c5cb4f2cdce/numpy/core/src/multiarray/arraytypes.c.src#L4403. Since we expect to be able to mutate a datetime64 dtype in libarrow_python, this can cause failures in other code when uses a dtype that we have accidentally mutated internally.

      Luckily we are only modifying dtype metadata in a couple of places so this should not be too invasive to fix: https://github.com/apache/arrow/blob/master/cpp/src/arrow/python/pandas_convert.cc#L1995

        Attachments

          Activity

            People

            • Assignee:
              wesm Wes McKinney
              Reporter:
              wesm Wes McKinney
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: