[ARROW-8164] [C++][Dataset] Let datasets be viewable with non-identical schema

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.16.0
• Fix Version/s: 0.17.0
• Component/s: C++
• Labels: dataset pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/24366
• Assignee: Ben Kietzman
• Reporter: Ben Kietzman
• Created: 19/Mar/20 15:55
• Updated: 11/Jan/23 07:58
• Resolved: 09/Apr/20 15:24

Description

It would be useful to allow some schema unification capability after discovery has completed. For example, if a FileSystemDataset is being wrapped into a UnionDataset with another and their schemas are unifiable then there is no reason we can't create the UnionDataset (rather than emitting an error because the schemas are not identical).

I think this behavior will be most naturally expressed in C++ like so:

virtual Result<Dataset> Dataset::ReplaceSchema(std::shared_ptr<Schema> schema) const = 0;

which will raise an error if the provided schema is not unifiable with the current dataset schema.

If this needs to be extended to non trivial projections then this will probably warrant a separate class, ProjectedDataset or so. Definitely follow up material (if desired)

Issue Links

• supercedes: ARROW-7945 [C++][Dataset] Implement InMemoryDatasetFactory
• links to: GitHub Pull Request #6721