Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
ARROW-13611 adds a backpressure mechanism that works for unordered scans. However, this backpressure is not properly applied on ordered (i.e. ScanBatches and not ScanBatchedUnordered) scans.
The fix will be to modify the merge generator used on ordered scans so that, while it still will read ahead somewhat on several files, it will never deliver batches except from the currently read file.
Attachments
Issue Links
- links to