[ARROW-16523] [C++] Move ExecPlan scheduling into the plan

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 9.0.0
• Component/s: C++
• Labels: pull-request-available query-engine
• External issue URL: https://github.com/apache/arrow/issues/31884
• Assignee: Sasha Krassovsky
• Reporter: Weston Pace
• Created: 11/May/22 02:10
• Updated: 11/Jan/23 11:44
• Resolved: 14/Jul/22 23:11

Description

Source nodes and pipeline breakers need to schedule new thread tasks. These tasks run entire fused pipelines (e.g. the thread task could be thought of as analogous to a "driver" in some other models).

At the moment every node that needs to schedule tasks (scan node, hash-join node, aggregate node, etc.) handles this independently. The result is a lot of similar looking code and bugs like ARROW-15221 where one node takes care of cleanup but another doesn't.

We can centralize this by moving this scheduling into the ExecPlan itself and giving nodes an ability to schedule tasks via the ExecPlan.

Issue Links

• links to: GitHub Pull Request #13143