[ARROW-7965] [Python] Refine higher level dataset API

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.17.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/24184
• Assignee: Krisztian Szucs
• Reporter: Krisztian Szucs
• Created: 28/Feb/20 16:49
• Updated: 11/Jan/23 07:57
• Resolved: 14/Apr/20 22:52

Description

Provide a more intuitive way to construct nested dataset:

```python

1. instead of using confusing factory function
   dataset([
   factory("s3://old-taxi-data", format="parquet"),
   factory("local/path/to/new/data", format="csv")
   ])

1. let the user to construct a new dataset directly from dataset objects
   dataset([
   dataset("s3://old-taxi-data", format="parquet"),
   dataset("local/path/to/new/data", format="csv")
   ])
   ```

In the future we might want to introduce a new Dataset class which wraps functionality of both the dataset actory and the materialized dataset enabling optimizations over rediscovery of already materialized datasets.

Issue Links

• links to: GitHub Pull Request #6505