Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-1328

[Python] pyarrow.Table.from_pandas option timestamps_to_ms changes column values

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.5.0
    • 0.6.0
    • Python
    • None
    • MacOS 10.12.6
      python 2.7.12

      pyarrow==0.5.0
      pandas==0.20.3
      numpy==1.13.1
      python-dateutil==2.6.1

    Description

      The pyarrow.Table.from_pandas option timestamps_to_ms changes the value of the datetime column

      >>> import pyarrow as pa
      >>> import pandas as pd
      >>> from datetime import datetime
      >>>
      >>> df = pd.DataFrame({'datetime': [datetime(2017, 1, 1)]})
      >>> dtype = df['datetime'].dtype
      >>> pa_type = pa.from_numpy_dtype(dtype)
      >>> schema = pa.schema([pa.field('datetime', pa_type)])
      >>> table = pa.Table.from_pandas(df=df, schema=schema, timestamps_to_ms=True)
      >>> table.to_pandas()
                          datetime
      0 1970-01-01 00:24:43.228800
      
      >>> dtype
      dtype('<M8[ns]')
      >>> pa_type
      TimestampType(timestamp[ns])
      

      Attachments

        Activity

          People

            wesm Wes McKinney
            fjetter Florian Jetter
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: