[ARROW-15238] [C++] Create "engine" module for the query engine

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.0.0
• Component/s: C++
• Labels: pull-request-available query-engine
• External issue URL: https://github.com/apache/arrow/issues/30735
• Assignee: Jeroen van Straten
• Reporter: Weston Pace
• Created: 03/Jan/22 19:47
• Updated: 11/Jan/23 08:45
• Resolved: 16/Feb/22 02:22

Description

Circular dependencies are popping up in the query engine as the compute module is very low level. For example, it would be nice if the default registry included the scan node and dataset write node. We will want to be adding spillover support at some point and that will rely on parquet/dataset operations.

We should create a dedicated engine module which includes the query plans, the nodes, etc. This module would not contain the kernels or other low level compute primitives. This way we could have something like...

engine -> datasets (for scanning) -> parquet -> compute (for calculating statistics)

The base ExecPlan itself could either go in compute or engine depending on which has the least amount of friction.

Issue Links

• is depended upon by: ARROW-15257 [C++] Simplify ExecPlan's C++ interface
• links to: GitHub Pull Request #11707
• links to: GitHub Pull Request #12279