[ARROW-16872] [Python] open_csv throws ArrowInvalid if csv does not end with a new line and is above 16384 lines

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 7.0.0, 8.0.0
• Fix Version/s: 9.0.0
• Component/s: Python
• Labels: csvparser open_csv pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/32196

Language

Python

• Assignee: Yibo Cai
• Reporter: Frederik Fabritius
• Created: 21/Jun/22 19:25
• Updated: 11/Jan/23 11:47
• Resolved: 30/Jun/22 16:31

Description

`pyarrow.csv.open_csv` throws ArrowInvalid if csv does not end with a new line and is above 16384 lines. Tested with both pyarrow 7.0.0 and 8.0.0. Error seen both in production app and on developer laptop.

 

Here's a minimal case for reproducing the issue:

```python

import pyarrow as pa

import pyarrow.csv

from io import BytesIO

for _ in pa.csv.open_csv(BytesIO('\n'.join(['review_id,filter_outcome'] + ['62593aaec7628b203bad4c6e,fabrication']*16385).encode())): pass

```

 

Error is thrown: 

ArrowInvalid: CSV parse error: Expected 2 columns, got 1: 

Issue Links

• links to: GitHub Pull Request #13437