[ARROW-10146] [Python] Parquet metadata to_dict raises attribute error

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/26157
• Assignee: Joris Van den Bossche
• Reporter: Florian Jetter
• Created: 30/Sep/20 16:13
• Updated: 11/Jan/23 08:11
• Resolved: 07/Dec/20 16:33

Description

Description

When accessing rowgroup metadata and trying to convert it to a dict using the method to_dict I sometimes receive an Attribute error.

This can be consistently produced with an empty dataframe (see example below) but I have also seen it already for non empty dataframes. I couldn't track down what makes the non-empty Attribute errors special, therefore the example below.

Expected behaviour

I would expect the to_dict to always, consistently return a dictionary with the appropriate metadata and statistics irregardless of the file content.

Minimal Example

 

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

df = pd.DataFrame({"col": [1]}).head(0)
table = pa.Table.from_pandas(df)
buf = pa.BufferOutputStream()
pq.write_table(table, buf)
reader = pa.BufferReader(buf.getvalue())
parquet_file = pq.ParquetFile(reader)
# Raises Attribute Error
parquet_file.metadata.to_dict()

Traceback

~/miniconda3/envs/kartothek-dev/lib/python3.7/site-packages/pyarrow/_parquet.pyx in pyarrow._parquet.FileMetaData.to_dict()

~/miniconda3/envs/kartothek-dev/lib/python3.7/site-packages/pyarrow/_parquet.pyx in pyarrow._parquet.RowGroupMetaData.to_dict()

~/miniconda3/envs/kartothek-dev/lib/python3.7/site-packages/pyarrow/_parquet.pyx in pyarrow._parquet.ColumnChunkMetaData.to_dict()

AttributeError: 'NoneType' object has no attribute 'to_dict'

Versions

In [28]: pa.__version__
Out[28]: '1.0.1'

In [29]: pd.__version__
Out[29]: '1.0.5'

Issue Links

• links to: GitHub Pull Request #8861