Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-1274

[Python] SegFault in pyarrow.parquet.write_table with specific options

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: cpp-1.5.0
    • Component/s: parquet-cpp
    • Environment:

      tested on MacOS High Sierra with python 3.6 and Ubuntu Xenial (Python 3.5)

      Description

      I originally filed an issue in the pandas project but we've tracked it down to arrow itself, when called via pandas in specific circumstances:

      https://github.com/pandas-dev/pandas/issues/19493

      basically using

       df.to_parquet('filename.parquet', flavor='spark')

      gives a seg fault if `df` contains a datetime column.

      Under the covers,  pandas translates this to the following call:

      pq.write_table(table, 'output.parquet', flavor='spark', compression='snappy', coerce_timestamps='ms')
      

      which gives me an instant crash.

      There is a repro on the github ticket.

       

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                joshuastorck Joshua Storck
                Reporter:
                clembou Clément Bouscasse
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: