Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-14191

[C++][Dataset] Dataset writes should respect backpressure

Details

    Description

      If the write destination is slow then the dataset writer should back off and apply backpressure to pause the reader. This will allow simple dataset API scans to operate on large out of core datasets.

      This is dependent on ARROW-13611 which adds a backpressure feature for regular scanning (but not writing data) and on ARROW-13542 which moves the dataset write to a node in the exec plan.

      Attachments

        Issue Links

          Activity

            People

              westonpace Weston Pace
              westonpace Weston Pace
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 10m
                  2h 10m

                  Slack

                    Issue deployment