[ARROW-2579] [Python] Appending to streamable table file format doesn't seem to work

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.0
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/18955
• Assignee: Unassigned
• Reporter: Rob Ambalu
• Created: 14/May/18 15:23
• Updated: 11/Jan/23 07:21

Description

As far as I can tell it looks like appending to a streaming file format isn’t currently supported, is that right?

RecordBatchStreamWriter always writes the schema up front, and it doesn’t look like a schema is expected mid file ( assuming im doing this append test correctly, this is the error I hit when I try to read back this file into python:

 Traceback (most recent call last):

  File "/home/ra7293/rba_arrow_mmap.py", line 9, in <module>

    table = reader.read_all()

  File "ipc.pxi", line 302, in pyarrow.lib._RecordBatchReader.read_all

  File "error.pxi", line 79, in pyarrow.lib.check_status

pyarrow.lib.ArrowIOError: Message not expected type: record batch, was: 1

 

This reader script works fine if I write once / don’t append.

Seeing as IO interfaces support Append, streaming should support it as well ( if for whatever reason this cant be supported, RecordBatchStreamWriter should throw if configured with an OutputStreamer that is attempting to append )