[ARROW-2369] Large (>~20 GB) files written to Parquet via PyArrow are corrupted

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.10.0
• Component/s: Python
• Labels: Parquet bug pandas parquetWriter pull-request-available pyarrow
• Environment: Reproduced on Ubuntu + Mac OSX
• External issue URL: https://github.com/apache/arrow/issues/18367
• Assignee: Antoine Pitrou
• Reporter: Justin Tan
• Created: 30/Mar/18 13:00
• Updated: 11/Jan/23 07:20
• Resolved: 12/Apr/18 12:11

Description

When writing large Parquet files (above 10 GB or so) from Pandas to Parquet via the command

pq.write_table(my_df, 'table.parquet')

The write succeeds, but when the parquet file is loaded, the error message

ArrowIOError: Invalid parquet file. Corrupt footer.

appears. This same error occurs when the parquet file is written chunkwise as well. When the parquet files are small, say < 5 GB or so (drawn randomly from the same dataset), everything proceeds as normal. I've also tried this with Pandas df.to_parquet(), with the same results.

Update: Looks like any DataFrame with size above ~5GB (on disk) returns the same error.

Issue Links

• supercedes: ARROW-2372 [Python] ArrowIOError: Invalid argument when reading Parquet file
• links to: GitHub Pull Request #1866