Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-8115

[Python] Conversion when mixing NaT and datetime objects not working

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.0.0
    • Python
    • None

    Description

      Code Sample

      import pandas as pd
      df = pd.DataFrame({"date": ["", "2019-05-01"]})
      df.date = pd.to_datetime(df.date).dt.date
      df.to_parquet("issue_NaT_parquet")
      

      Problem description

      The above gives me the following error:

      ---------------------------------------------------------------------------
      TypeError                                 Traceback (most recent call last)
      <ipython-input-13-432405bef6ac> in <module>
      ----> 1 df.to_parquet("test.parquet")
      
      ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pandas\util\_decorators.py in wrapper(*args, **kwargs)
          212                 else:
          213                     kwargs[new_arg_name] = new_arg_value
      --> 214             return func(*args, **kwargs)
          215
          216         return cast(F, wrapper)
      
      ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pandas\core\frame.py in to_parquet(self, path, engine, compression, index, partition_cols, **kwargs)
         2114             index=index,
         2115             partition_cols=partition_cols,
      -> 2116             **kwargs,
         2117         )
         2118
      
      ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pandas\io\parquet.py in to_parquet(df, path, engine, compression, index, partition_cols, **kwargs)
          262         index=index,
          263         partition_cols=partition_cols,
      --> 264         **kwargs,
          265     )
          266
      
      ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pandas\io\parquet.py in write(self, df, path, compression, coerce_timestamps, index, partition_cols, **kwargs)
           99             from_pandas_kwargs["preserve_index"] = index
          100
      --> 101         table = self.api.Table.from_pandas(df, **from_pandas_kwargs)
          102         if partition_cols is not None:
          103             self.api.parquet.write_to_dataset(
      
      ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\table.pxi in pyarrow.lib.Table.from_pandas()
      
      ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\pandas_compat.py in dataframe_to_arrays(df, schema, preserve_index, nthreads, columns, safe)
          573     if nthreads == 1:
          574         arrays = [convert_column(c, f)
      --> 575                   for c, f in zip(columns_to_convert, convert_fields)]
          576     else:
          577         from concurrent import futures
      
      ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\pandas_compat.py in <listcomp>(.0)
          573     if nthreads == 1:
          574         arrays = [convert_column(c, f)
      --> 575                   for c, f in zip(columns_to_convert, convert_fields)]
          576     else:
          577         from concurrent import futures
      
      ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\pandas_compat.py in convert_column(col, field)
          558
          559         try:
      --> 560             result = pa.array(col, type=type_, from_pandas=True, safe=safe)
          561         except (pa.ArrowInvalid,
          562                 pa.ArrowNotImplementedError,
      
      ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\array.pxi in pyarrow.lib.array()
      
      ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\array.pxi in pyarrow.lib._ndarray_to_array()
      
      TypeError: an integer is required (got type datetime.date){quote}
      

      Expected Output

      Parquet with null values mixed with date values.

      Attachments

        Issue Links

          Activity

            People

              wesm Wes McKinney
              IlyaOrson Ilya Orson Sandoval
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: