Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-3652

[Python] CategoricalIndex is lost after reading back

    XMLWordPrintableJSON

    Details

      Description

      When a CategoricalIndex is written and read back the resulting index is not more categorical.

      df = pd.DataFrame([['a', 'b'], ['c', 'd']], columns=['c1', 'c2'])
      df['c1'] = df['c1'].astype('category')
      df = df.set_index(['c1'])
      
      table = pa.Table.from_pandas(df)
      pq.write_table(table, 'test.parquet')
      
      ref_df = pq.read_pandas('test.parquet').to_pandas()
      
      print(df.index)
      # CategoricalIndex(['a', 'c'], categories=['a', 'c'], ordered=False, name='c1', dtype='category')
      
      print(ref_df.index)
      # Index(['a', 'c'], dtype='object', name='c1')
      

      In the metadata the information is correctly contained:

      {"name": "c1", "field_name": "c1", "p'
                  b'andas_type": "categorical", "numpy_type": "int8", "metadata": {"'
                  b'num_categories": 2, "ordered": false}
      

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                wesm Wes McKinney
                Reporter:
                aberres Armin Berres
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h