Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-2122

[Python] Pyarrow fails to serialize dataframe with timestamp.

    XMLWordPrintableJSON

    Details

      Description

      The bug can be reproduced as follows.

      import pyarrow as pa
      import pandas as pd
      
      df = pd.DataFrame({'A': [pd.Timestamp('2012-11-11 00:00:00+01:00'), pd.NaT]}) 
      
      s = pa.serialize(df).to_buffer()
      new_df = pa.deserialize(s) # this fails

      The last line fails with

      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "serialization.pxi", line 441, in pyarrow.lib.deserialize
        File "serialization.pxi", line 404, in pyarrow.lib.deserialize_from
        File "serialization.pxi", line 257, in pyarrow.lib.SerializedPyObject.deserialize
        File "serialization.pxi", line 174, in pyarrow.lib.SerializationContext._deserialize_callback
        File "/home/ubuntu/arrow/python/pyarrow/serialization.py", line 77, in _deserialize_pandas_dataframe
          return pdcompat.serialized_dict_to_dataframe(data)
        File "/home/ubuntu/arrow/python/pyarrow/pandas_compat.py", line 446, in serialized_dict_to_dataframe
          for block in data['blocks']]
        File "/home/ubuntu/arrow/python/pyarrow/pandas_compat.py", line 446, in <listcomp>
          for block in data['blocks']]
        File "/home/ubuntu/arrow/python/pyarrow/pandas_compat.py", line 466, in _reconstruct_block
          dtype = _make_datetimetz(item['timezone'])
        File "/home/ubuntu/arrow/python/pyarrow/pandas_compat.py", line 481, in _make_datetimetz
          return DatetimeTZDtype('ns', tz=tz)
        File "/home/ubuntu/anaconda3/lib/python3.5/site-packages/pandas/core/dtypes/dtypes.py", line 409, in __new__
          raise ValueError("DatetimeTZDtype constructor must have a tz "
      ValueError: DatetimeTZDtype constructor must have a tz supplied

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                adshieh Albert Shieh
                Reporter:
                robertnishihara Robert Nishihara
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: