[ARROW-10617] [Python] RecordBatchStreamReader's iterator doesn't work with python 3.8

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.1
• Fix Version/s: 3.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/26574
• Assignee: Tao He
• Reporter: Tao He
• Created: 16/Nov/20 13:07
• Updated: 11/Jan/23 08:14
• Resolved: 07/Dec/20 18:28

Description

The following example code doesn't work with python 3.8:

import pyarrow as pa
data = [
    pa.array([1, 2, 3, 4]),
    pa.array(['foo', 'bar', 'baz', None]),
    pa.array([True, None, False, True])
]
batch = pa.record_batch(data, names=['f0', 'f1', 'f2'])

sink = pa.BufferOutputStream()
writer = pa.ipc.new_stream(sink, batch.schema)

for i in range(5):
    writer.write_batch(batch)
writer.close()

buf = sink.getvalue()

reader = pa.ipc.open_stream(buf)
[i for i in reader]

It will raise the following runtime error:

StopIteration Traceback (most recent call last)
{{ /usr/local/lib/python3.8/dist-packages/pyarrow/ipc.pxi in pyarrow.lib._CRecordBatchReader.read_next_batch()}}

StopIteration:

During handling of the above exception, another exception occurred:

RuntimeError Traceback (most recent call last)
{{ <ipython-input-7-225bed213dc7> in <module>}}
{{ 18 reader = pa.ipc.open_stream(buf)}}
{{ 19}}
{{ ---> 20 [i for i in reader]}}

<ipython-input-7-225bed213dc7> in <listcomp>(.0)
{{ 18 reader = pa.ipc.open_stream(buf)}}
{{ 19}}
{{ ---> 20 [i for i in reader]}}

/usr/local/lib/python3.8/dist-packages/pyarrow/ipc.pxi in _iter_()

RuntimeError: generator raised StopIteration

Issue Links

• links to: GitHub Pull Request #8677