[ARROW-1328] [Python] pyarrow.Table.from_pandas option timestamps_to_ms changes column values

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.5.0
• Fix Version/s: 0.6.0
• Component/s: Python
• Environment: MacOS 10.12.6; python 2.7.12; ; pyarrow==0.5.0; pandas==0.20.3; numpy==1.13.1; python-dateutil==2.6.1
• External issue URL: https://github.com/apache/arrow/issues/17362
• Assignee: Wes McKinney
• Reporter: Florian Jetter
• Created: 03/Aug/17 15:31
• Updated: 11/Jan/23 07:14
• Resolved: 04/Aug/17 18:26

Description

The pyarrow.Table.from_pandas option timestamps_to_ms changes the value of the datetime column

>>> import pyarrow as pa
>>> import pandas as pd
>>> from datetime import datetime
>>>
>>> df = pd.DataFrame({'datetime': [datetime(2017, 1, 1)]})
>>> dtype = df['datetime'].dtype
>>> pa_type = pa.from_numpy_dtype(dtype)
>>> schema = pa.schema([pa.field('datetime', pa_type)])
>>> table = pa.Table.from_pandas(df=df, schema=schema, timestamps_to_ms=True)
>>> table.to_pandas()
                    datetime
0 1970-01-01 00:24:43.228800

>>> dtype
dtype('<M8[ns]')
>>> pa_type
TimestampType(timestamp[ns])