Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
None
Description
Nightly integration tests with kartothek are failing, see eg https://github.com/ursacomputing/crossbow/runs/4863725914?check_suite_focus=true
This seems something on our side, and a recent failure (the builds only started failing today, and I don't see other differences with the last working build yesterday)
Update, a reproducer:
In [4]: df = pd.DataFrame({'a': [1, 2], 'b': [0.1, 0.2]})
In [5]: table = pa.table(df)
In [6]: table.schema.empty_table().to_pandas()
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-6-a03ecffc0af8> in <module>
----> 1 table.schema.empty_table().to_pandas()
~/scipy/repos/arrow/python/pyarrow/array.pxi in pyarrow.lib._PandasConvertible.to_pandas()
~/scipy/repos/arrow/python/pyarrow/table.pxi in pyarrow.lib.Table._to_pandas()
~/scipy/repos/arrow/python/pyarrow/pandas_compat.py in table_to_blockmanager(options, table, categories, ignore_metadata, types_mapper)
790
791 axes = [columns, index]
--> 792 return BlockManager(blocks, axes)
793
794
~/miniconda3/envs/arrow-dev/lib/python3.8/site-packages/pandas/core/internals/managers.py in __init__(self, blocks, axes, verify_integrity)
912 pass
913
--> 914 self._verify_integrity()
915
916 def _verify_integrity(self) -> None:
~/miniconda3/envs/arrow-dev/lib/python3.8/site-packages/pandas/core/internals/managers.py in _verify_integrity(self)
919 for block in self.blocks:
920 if block.shape[1:] != mgr_shape[1:]:
--> 921 raise construction_error(tot_items, block.shape[1:], self.axes)
922 if len(self.items) != tot_items:
923 raise AssertionError(
ValueError: Empty data passed with indices specified.
It happens specifically if the schema still has pandas metadata that indicate a range for the index (which we try to recreate, but that doesn't match the actual length of the table).
Attachments
Issue Links
- links to
