[ARROW-14191] [C++][Dataset] Dataset writes should respect backpressure

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 6.0.0
• Component/s: C++
• Labels: pull-request-available query-engine
• External issue URL: https://github.com/apache/arrow/issues/29776
• Assignee: Weston Pace
• Reporter: Weston Pace
• Created: 01/Oct/21 11:23
• Updated: 11/Jan/23 08:38
• Resolved: 15/Oct/21 20:31

Description

If the write destination is slow then the dataset writer should back off and apply backpressure to pause the reader. This will allow simple dataset API scans to operate on large out of core datasets.

This is dependent on ARROW-13611 which adds a backpressure feature for regular scanning (but not writing data) and on ARROW-13542 which moves the dataset write to a node in the exec plan.

Issue Links

• depends upon: ARROW-13611 [C++] Scanning datasets does not enforce back pressure
• depends upon: ARROW-13542 [C++][Compute][Dataset] Add dataset::WriteNode for writing rows from an ExecPlan to disk
• links to: GitHub Pull Request #11286