[ARROW-9958] Error writing record batches to IPC streaming format

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.0.1
• Component/s: GLib, Python
• Labels: Compression
• Environment: pyarrow - Version: 1.0.1 ; python - version 3.7.6 ; Operating system - CentOS Linux release 7.8.2003 (Core)
• External issue URL: https://github.com/apache/arrow/issues/25984
• Assignee: Unassigned
• Reporter: Ishan
• Created: 10/Sep/20 07:13
• Updated: 11/Jan/23 08:10
• Resolved: 10/Sep/20 21:10

Description

Writing record batches to the Arrow IPC streaming format with on-the-fly compression generally raises errors of one type or the other when reading it back.

PFA the code producing each of the below errors. I can't reproduce it for smaller batch sizes, so it probably has to do with size of each record batch. It does not seem specific to pyarrow since I see a similar issue with the C-Glib API.

#Error case 1

```
~/py376/lib/python3.7/site-packages/pyarrow/ipc.pxi in pyarrow.lib._CRecordBatchReader.read_next_batch()

~/py376/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

OSError: Truncated compressed stream
```

#Error case 2

```
~/py376/lib/python3.7/site-packages/pyarrow/ipc.pxi in pyarrow.lib._RecordBatchStreamReader._open()

~/py376/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/py376/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Tried reading schema message, was null or length 0
```