Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-14192

[C++][Dataset] Backpressure broken on ordered scans

    XMLWordPrintableJSON

Details

    Description

      ARROW-13611 adds a backpressure mechanism that works for unordered scans. However, this backpressure is not properly applied on ordered (i.e. ScanBatches and not ScanBatchedUnordered) scans.

      The fix will be to modify the merge generator used on ordered scans so that, while it still will read ahead somewhat on several files, it will never deliver batches except from the currently read file.

      Attachments

        Issue Links

          Activity

            People

              westonpace Weston Pace
              westonpace Weston Pace
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 50m
                  2h 50m