[ARROW-3098] [Python] BufferReader doesn't adhere to the seek protocol

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 0.11.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/19454
• Assignee: Antoine Pitrou
• Reporter: Björn Andersson
• Created: 21/Aug/18 09:14
• Updated: 11/Jan/23 07:25
• Resolved: 21/Aug/18 16:20

Description

I have a script that creates a Parquet file and then writes it out to a BufferOutputStream and then into a BufferReader with the intention of passing it to a place that takes a file-like object to upload it somewhere else. But the other location relies on being able to seek to the end of the file to figure out how big the file is, e.g.

reader.seek(0, 2)
size = reader.tell()
reader.seek(0)

 

But when I do that the following exception is raised: 

 

pyarrow/io.pxi:209: in pyarrow.lib.NativeFile.seek
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

> ???
E pyarrow.lib.ArrowIOError: position out of bounds

I compared it to casting to an io.BytesIO instead which works:

import io

import pyarrow as pa


def test_arrow_output_stream():
    output = pa.BufferOutputStream()
    output.write(b'hello')

    reader = pa.BufferReader(output.getvalue())

    reader.seek(0, 2)
    assert reader.tell() == 5


def test_python_io_stream():
    output = pa.BufferOutputStream()
    output.write(b'hello')

    buffer = io.BytesIO(output.getvalue().to_pybytes())
    reader = io.BufferedRandom(buffer)

    reader.seek(0, 2)
    assert reader.tell() == 5

Issue Links

• links to: GitHub Pull Request #2454