[ARROW-25] [C++] Implement delimited file scanner / CSV reader

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Component/s: C++
• Labels: csv pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/15374
• Assignee: Antoine Pitrou
• Reporter: Wes McKinney
• Created: 02/Mar/16 19:40
• Updated: 11/Jan/23 07:06
• Resolved: 01/Oct/18 10:30

Description

Like Parquet and binary file formats, text files will be an important data medium for converting to and from in-memory Arrow data.

pandas has some (Apache-compatible) business logic we can learn from here (as one of the gold-standard CSV readers in production use)

https://github.com/pydata/pandas/blob/master/pandas/src/parser/tokenizer.h
https://github.com/pydata/pandas/blob/master/pandas/parser.pyx

While very fast, this this should be largely written from scratch to target the Arrow memory layout, but we can reuse certain aspects like the tokenizer DFA (which originally came from the Python interpreter csv module implementation)

https://github.com/pydata/pandas/blob/master/pandas/src/parser/tokenizer.c#L713

Issue Links

• links to: GitHub Pull Request #2576