[ARROW-4333] [C++] Sketch out design for kernels and "query" execution in compute layer

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 6.0.0
• Component/s: C++
• Labels: analytics
• External issue URL: https://github.com/apache/arrow/issues/20902
• Assignee: Unassigned
• Reporter: Micah Kornfield
• Created: 23/Jan/19 03:40
• Updated: 11/Jan/23 07:33
• Resolved: 29/Aug/22 14:14

Description

It would be good to formalize the design of kernels and the controlling query execution layer (e.g. volcano batch model?) to understand the following:

Contracts for kernels:

• Thread safety of kernels?
• When Kernels should allocate memory vs expect preallocated memory?  How to communicate requirements for a kernels memory allocaiton?
• How to communicate the whether a kernels execution is parallelizable across a ChunkedArray?  How to determine if the order to execution across a ChunkedArray is important?
• How to communicate when it is safe to re-use the same buffers and input and output to the same kernel?

What does the threading model look like for the higher level of control?  Where should synchronization happen?

Issue Links

• relates to: ARROW-8894 [C++] C++ array kernels framework and execution buildout (umbrella issue)