[ARROW-17229] [C++] ReadRel is translated to a source node that emits unexpected fields

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: C++
• Labels: query-engine substrait
• External issue URL: https://github.com/apache/arrow/issues/32516
• Assignee: Unassigned
• Reporter: David Li
• Created: 27/Jul/22 18:20
• Updated: 11/Jan/23 11:49

Description

Currently, a Substrait plan with a RelRoot containing a ReadRel will contain extra, unexpected fields, namely __fragment_index et. al. Right now they are always included by default. There are a few things to be done:

• ReadRel's base_schema could be converted into a ScanOptions.dataset_schema to limit the fields read. (Also see ARROW-15585, these fields should be used for pushdown projection)
• The scanner always adds these extra fields - maybe it should be opt-in instead
• There's no way to manually insert a Project to "fix" things because as implemented, it can only add new columns

Issue Links

• relates to: ARROW-15585 [C++] Add support for Substrait pushdown projection