[ARROW-8065] [C++][Dataset] Untangle Dataset, Fragment and ScanOptions

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: C++
• Labels: dataset pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/24278
• Assignee: Francois Saint-Jacques
• Reporter: Francois Saint-Jacques
• Created: 10/Mar/20 19:03
• Updated: 11/Jan/23 07:57
• Resolved: 23/Apr/20 18:13

Description

Currently: a fragment is a product of a scan; it is a lazy collection of scan tasks corresponding to a data source which is logically singular (like a single file, a single row group, ...). It would be more useful if instead a fragment were the direct object of a scan; one scans a fragment (or a collection of fragments):

1. Remove ScanOptions from Fragment's properties and move it into Fragment::Scan parameters.
2. Remove ScanOptions from Dataset::GetFragments. We can provide an overload to support predicate pushdown in FileSystemDataset and UnionDataset Dataset::GetFragments(std::shared_ptr<Expression> predicate).
3. Expose lazy accessor to Fragment::physical_schema()
4. Consolidate ScanOptions and ScanContext

This will lessen the cognitive dissonance between fragments and files since fragments will no longer include references to scan properties.

Issue Links

• blocks: ARROW-8282 [C++/Python][Dataset] Support schema evolution for integer columns
• blocks: ARROW-8318 [C++][Dataset] Dataset should instantiate Fragment
• links to: GitHub Pull Request #7000