[ARROW-17583] [Python] File write visitor throws exception on large parquet file

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 9.0.0
• Fix Version/s: 10.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/32831
• Assignee: Joost Hoozemans
• Reporter: Joost Hoozemans
• Created: 31/Aug/22 15:43
• Updated: 11/Jan/23 11:51
• Resolved: 08/Sep/22 08:36

Description

When writing a large parquet file (e.g. 5GB) using pyarrow.dataset, it throws an exception:

Traceback (most recent call last):
  File "pyarrow/_dataset_parquet.pyx", line 165, in pyarrow._dataset_parquet.ParquetFileFormat._finish_write
  File "pyarrow/dataset.pyx", line 2695, in pyarrow._dataset.WrittenFile.init_
OverflowError: value too large to convert to int
Exception ignored in: 'pyarrow._dataset._filesystemdataset_write_visitor'

The file is written succesfully though. It seems related to this issue https://issues.apache.org/jira/browse/ARROW-16761.

I would guess the problem is the python field is an int while the C++ code returns an int64_t https://github.com/apache/arrow/pull/13338/files#diff-4f2eb12337651b45bab2b03abe2552dd7fc9958b1fbbeb09a2a488804b097109R164 

Issue Links

• links to: GitHub Pull Request #14032