[ARROW-16036] [C++] Substrait SinkNode Modification for usability

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: C++
• Labels: query-engine substrait
• External issue URL: https://github.com/apache/arrow/issues/31455
• Assignee: Unassigned
• Reporter: Vibhatha Lakmal Abeykoon
• Created: 26/Mar/22 03:27
• Updated: 11/Jan/23 11:41

Description

Currently, the Substrait implementation uses a `consuming_sink` node as the end-point to capture the results of an executed query. To further enhance the ability to use the wide variety of `SinkNode`s in the streaming execution engine, it is better to evaluate other options to provide usability for various use cases. 

Some of the use cases include consuming a Substrait query plan in R and Python APIs. And this extends beyond the Arrow APIs where third-party tools willing to use Substrait can be benefitted from these. At the moment there is active work on providing R/Python APIs for Substrait consumption (https://issues.apache.org/jira/browse/ARROW-15779). 
A potential modification to provide usability is to take in a `SinkNodeFactory`-like interface instead of just using the `ConsumingSink` (current implementation). Such a modification would provide flexibility in developing a wide array of applications. 
This issue is open for discussion.