[ARROW-6341] [Python] Implement low-level bindings for Dataset

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Component/s: Python
• Labels: dataset pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/22718
• Assignee: Krisztian Szucs
• Reporter: Francois Saint-Jacques
• Created: 23/Aug/19 21:18
• Updated: 11/Jan/23 07:46
• Resolved: 13/Dec/19 17:26

Description

The following classes should be accessible from Python:

• class DataSource
• class DataSourceDiscovery
• class Dataset
• class ScanContext, ScanOptions, ScanTask
• class ScannerBuilder
• class Scanner

The end result is reading a directory of parquet files as a single stream. One should be able to re-implement https://github.com/apache/arrow/pull/5720 in python.

Issue Links

• depends upon: ARROW-6242 [C++] Implements basic Dataset/Scanner/ScannerBuilder
• links to: GitHub Pull Request #5237