[ARROW-15116] [Python] Expose invalid_row_handler for CSV reader

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 6.0.1
• Fix Version/s: 7.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/30626
• Assignee: Antoine Pitrou
• Reporter: Bin Sun
• Created: 15/Dec/21 15:29
• Updated: 11/Jan/23 08:44
• Resolved: 29/Dec/21 16:18

Description

I deal with 10G+ csv files (tab separated) every day. However, rows are of 21 or 22 fields uncertainly.

If using pyarrow.csv.read_csv(),  error occurs.  I see ParseOptions(invalid_row_handler=...) in C++ API, but not in python API.

Is there a due time for this api in python? Thanks in advance!

Currently, I have to use pandas.read_csv(on_bad_lines=...) , which is extremely slow for 10G+ files...  

Issue Links

• causes: ARROW-15234 [Python] Possible crash with custom CSV invalid row handler
• links to: GitHub Pull Request #12012