[PARQUET-1273] [Python] Error writing to partitioned Parquet dataset

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: cpp-1.5.0
• Component/s: parquet-cpp
• Labels: pull-request-available

Environment

Linux (Ubuntu 16.04)

• Assignee: Joshua Storck
• Reporter: Robert Dailey
• Created: 19/Dec/17 17:22
• Updated: 23/Jun/24 03:30
• Resolved: 18/Apr/18 08:10

Description

I receive the following error after upgrading to pyarrow 0.8.0 when writing to a dataset:

• ArrowIOError: Column 3 had 187374 while previous column had 10000

The command was:
write_table_values =

{'row_group_size': 10000}

pq.write_to_dataset(pa.Table.from_pandas(df, preserve_index=True), '/logs/parsed/test', partition_cols=['Product', 'year', 'month', 'day', 'hour'], **write_table_values)

I've also tried write_table_values =

{'chunk_size': 10000}

and received the same error.

This same command works in version 0.7.1. I am trying to troubleshoot the problem but wanted to submit a ticket.

Issue Links

• links to: GitHub Pull Request #453