[ARROW-11889] [C++] Add parallelism to streaming CSV reader

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 5.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/27731
• Assignee: Weston Pace
• Reporter: Weston Pace
• Created: 06/Mar/21 01:29
• Updated: 11/Jan/23 08:22
• Resolved: 16/Jul/21 21:52

Description

Currently the streaming CSV reader does not allow for much parallelism.  It doesn't allow for reading more than one segment at once (useful in S3) and it doesn't allow for column fan-out for parsing & converting.

It seems both of these options would speed up CSV reading in some scenarios although it's possible this is mostly mitigated in cases where there are many more files than cores (as per-file parallelism will occupy all the cores anyways).

Issue Links

• links to: GitHub Pull Request #10568