Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-1328

[Python] pyarrow.Table.from_pandas option timestamps_to_ms changes column values

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.6.0
    • Component/s: Python
    • Labels:
      None
    • Environment:
      MacOS 10.12.6
      python 2.7.12

      pyarrow==0.5.0
      pandas==0.20.3
      numpy==1.13.1
      python-dateutil==2.6.1

      Description

      The pyarrow.Table.from_pandas option timestamps_to_ms changes the value of the datetime column

      >>> import pyarrow as pa
      >>> import pandas as pd
      >>> from datetime import datetime
      >>>
      >>> df = pd.DataFrame({'datetime': [datetime(2017, 1, 1)]})
      >>> dtype = df['datetime'].dtype
      >>> pa_type = pa.from_numpy_dtype(dtype)
      >>> schema = pa.schema([pa.field('datetime', pa_type)])
      >>> table = pa.Table.from_pandas(df=df, schema=schema, timestamps_to_ms=True)
      >>> table.to_pandas()
                          datetime
      0 1970-01-01 00:24:43.228800
      
      >>> dtype
      dtype('<M8[ns]')
      >>> pa_type
      TimestampType(timestamp[ns])
      

        Attachments

          Activity

            People

            • Assignee:
              wesmckinn Wes McKinney
              Reporter:
              fjetter Florian Jetter
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: