[ARROW-14192] [C++][Dataset] Backpressure broken on ordered scans

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 6.0.0
• Component/s: C++
• Labels: pull-request-available query-engine
• External issue URL: https://github.com/apache/arrow/issues/29777
• Assignee: Weston Pace
• Reporter: Weston Pace
• Created: 01/Oct/21 11:27
• Updated: 11/Jan/23 08:38
• Resolved: 19/Oct/21 03:45

Description

ARROW-13611 adds a backpressure mechanism that works for unordered scans. However, this backpressure is not properly applied on ordered (i.e. ScanBatches and not ScanBatchedUnordered) scans.

The fix will be to modify the merge generator used on ordered scans so that, while it still will read ahead somewhat on several files, it will never deliver batches except from the currently read file.

Issue Links

• links to: GitHub Pull Request #11294