[ARROW-12386] [C++] Support file parallelism in AsyncScanner

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 5.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/28181
• Assignee: Weston Pace
• Reporter: Weston Pace
• Created: 14/Apr/21 16:54
• Updated: 11/Jan/23 08:26
• Resolved: 28/Apr/21 12:06

Description

Whether we pull from files in parallel or not is controlled by how we merge the batch streams in `AsyncScanner::ScanBatchesUnorderedAsync`.  Currently we are relying on `MakeConcatenatedGenerator` which is incorrect.  This is needed because `MakeMergedGenerator` pulls from its source (an `EnumeratingGenerator`) in an async reentrant fashion.  `MakeMergedGenerator` should not do this.  If some kind of readahead is truly necessary there then use `MakeReadaheadGenerator`.

Issue Links

• links to: GitHub Pull Request #10076