Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-1967

Python: AssertionError w.r.t Pandas conversion on Parquet files in 0.8.0 dev version

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: Python
    • Labels:
      None

      Description

      I have some Parquet files generated a few commit before 0.8.0 that I cannot load in 0.8.0.

      They have the following Pandas metadata:

      {u'pandas_version': u'0.20.3', u'index_columns': [u'__index_level_0__'], u'columns': [{u'pandas_type': u'unicode', u'name': u'col1', u'numpy_type': u'object', u'metadata': None}, {u'pandas_type': u'list[unicode]', u'name': u'partition', u'numpy_type': u'object', u'metadata': None}, {u'pandas_type': u'int64', u'name': None, u'numpy_type': u'int64', u'metadata': None}], u'column_indexes': [{u'pandas_type': u'unicode', u'name': None, u'numpy_type': u'object', u'metadata': None}]}
      

      And I get the following exception on loading the file:

          df = table.to_pandas()
        File "pyarrow/table.pxi", line 1043, in pyarrow.lib.Table.to_pandas (/home/jenkins/.conan/data/pyarrow/0.8.0/by/stable/build/0fda246af0df114df08c06ca4fdd2cac7d7696fd/arrow/python/build/temp.linux-x86_64-2.7/lib.cxx:46331)
        File "/io/venv_deb8/local/lib/python2.7/site-packages/pyarrow/pandas_compat.py", line 498, in table_to_blockmanager
          assert len(logical_index_names) == len(index_columns_set)
      AssertionError
      

        Attachments

          Activity

            People

            • Assignee:
              xhochy Uwe L. Korn
              Reporter:
              xhochy Uwe L. Korn
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: