Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-5895

[Python] New version stores timestamps as epoch ms instead of ISO timestamp string

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.14.0
    • Fix Version/s: None
    • Component/s: Python
    • Labels:
      None
    • Environment:
      Linux dev.office.whoop.com 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

      Description

      Just upgraded from pyarrow 0.13 to 0.14.

      Columns of type TimestampType(timestmap[ns]) now get written as epoch ms values: 
      1561939200507
      Where 0.13 wrote TimestampType(timestamp[ns]) as an ISO string:
      2019-07-01T00:00:00.507Z
      This broke my implementation.  How do I get pyarrow to write ISO strings again in 0.14?

       

      Here is my table write:

      {{ pyarrow.parquet.write_to_dataset(table=tbl, root_path=local_path,}}
      {{ partition_cols=['env', 'dt'],}}
      {{ coerce_timestamps='ms',}}
      {{ allow_truncated_timestamps=True,}}
      {{ version='2.0',}}
      {{ compression='SNAPPY')}}

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              johwilso1 John Wilson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: