[ARROW-17991] [Python] pyarrow.dataset IPC format does not support compression

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 11.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/33196
• Assignee: Joost Hoozemans
• Reporter: Joost Hoozemans
• Created: 11/Oct/22 19:32
• Updated: 11/Jan/23 11:57
• Resolved: 17/Oct/22 16:09

Description

When trying to write an IPC dataset using pyarrow.dataset, it is not possible to pass a compression argument:

Trying to pass a pyarrow.ipc.IpcWriteOptions object:

>>> ds.write_dataset(f, "./thing.arrow", format=ds.IpcFileFormat(), file_options=ipc.IpcWriteOptions(compression='lz4'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/joost/.cache/pypoetry/virtualenvs/datalogistik-rL_l_suP-py3.8/lib/python3.8/site-packages/pyarrow/dataset.py", line 940, in write_dataset
    if format != file_options.format:
AttributeError: 'pyarrow.lib.IpcWriteOptions' object has no attribute 'format'

 

Alternatively, pyarrow.dataset.IpcFileFormat().make_write_options() does not support a compression parameter

Issue Links

• links to: GitHub Pull Request #14400
• links to: GitHub Pull Request #14414