[ARROW-11981] [C++][Dataset][Compute] Replace UnionDataset with Union ExecNode

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 6.0.0
• Component/s: C++
• Labels: dataset kernel pull-request-available query-engine
• External issue URL: https://github.com/apache/arrow/issues/27813
• Assignee: Alexander Ocsa
• Reporter: Ben Kietzman
• Created: 16/Mar/21 13:17
• Updated: 11/Jan/23 08:23
• Resolved: 18/Aug/21 12:04

Description

UnionDataset allows Fragments of multiple schemas and differing file formats to be scanned together as a single Dataset. This is useful functionality but makes the Dataset interface somewhat difficult to reason about since it must be general enough to accommodate UnionDataset.

After ARROW-11928 it will probably be more natural to support unioning of datasets through a subclass of ExecNode. Reconciliation of differing schemas can then be trivially handled by a full ProjectNode.

Note this would obviate both ARROW-11001 and ARROW-11749. In addition, Dataset could be simplified to a concrete class containing a set of compatibly typed/formatted Fragments.

Issue Links

• is a child of: ARROW-12633 [C++] Query engine umbrella issue
• is related to: ARROW-13653 [C++][Dataset] Remove UnionDataset in favor of UnionExecNode
• links to: GitHub Pull Request #10927