[ARROW-5085] [Python/C++] Conversion of dict encoded null column fails in parquet writing when using RowGroups

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.13.0
• Fix Version/s: 0.15.0
• Component/s: C++
• Labels: parquet pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/21573
• Assignee: Wes McKinney
• Reporter: Florian Jetter
• Created: 01/Apr/19 15:54
• Updated: 11/Jan/23 07:38
• Resolved: 17/Aug/19 20:25

Description

Conversion of dict encoded null column fails in parquet writing when using RowGroups

import pyarrow.parquet as pq
import pandas as pd
import pyarrow as pa
df = pd.DataFrame({"col": [None] * 100, "int": [1.0] * 100})
df = df.astype({"col": "category"})
table = pa.Table.from_pandas(df)
buf = pa.BufferOutputStream()
pq.write_table(
    table,
    buf,
    version="2.0",
    chunk_size=10,
)

fails with

pyarrow.lib.ArrowIOError: Column 2 had 100 while previous column had 10

Issue Links

• links to: GitHub Pull Request #5107