[ARROW-3514] [Python] zlib deflate exception when writing Parquet file

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.11.0
• Fix Version/s: 0.11.1
• Component/s: C++, Python
• Labels: pull-request-available
• Environment: Amazon Linux, CentOS 7, Ubuntu 16.04, zlib 1.2.7/1.2.8, CPython 3.6.
• External issue URL: https://github.com/apache/arrow/issues/2756
• Assignee: Antoine Pitrou
• Reporter: Adam Machanic
• Created: 15/Oct/18 02:23
• Updated: 11/Jan/23 07:27
• Resolved: 16/Oct/18 11:33

Description

The below Python code throws an exception in 0.11.0, but not in 0.10.0.

I was able to reproduce the issue in Amazon Linux, CentOS 7, and Ubuntu 16.04, but not in Windows 7.

The Amazon and CentOS machines are both running zlib 1.2.7, and the Ubuntu machine is using 1.2.8.

Tested with CPython 3.6 in all cases.

import io
import pyarrow
from pyarrow import parquet

tbl = pyarrow.Table.from_arrays([pyarrow.array(['abc', 'def'])], ['some_col'])

f = io.BytesIO()
parquet.write_table(tbl, f, compression='gzip')

Following is the exception:

Traceback (most recent call last):
  File "test_pyarrow.py", line 8, in <module>
    parquet.write_table(tbl, f, compression='gzip')
  File "/home/adam/anaconda3/lib/python3.6/site-packages/pyarrow/parquet.py", line 1125, in write_table
    writer.write_table(table, row_group_size=row_group_size)
  File "/home/adam/anaconda3/lib/python3.6/site-packages/pyarrow/parquet.py", line 376, in write_table
    self.writer.write_table(table, row_group_size=row_group_size)
  File "pyarrow/_parquet.pyx", line 934, in pyarrow._parquet.ParquetWriter.write_table
  File "pyarrow/error.pxi", line 83, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Arrow error: IOError: zlib deflate failed, output buffer too small

Issue Links

• links to: GitHub Pull Request #2771