[ARROW-8286] [Python] Creating dataset from pathlib results in UnionDataset instead of FileSystemDataset

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.17.0
• Component/s: Python
• Labels: dataset pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/24480
• Assignee: Joris Van den Bossche
• Reporter: Joris Van den Bossche
• Created: 31/Mar/20 13:48
• Updated: 11/Jan/23 07:59
• Resolved: 31/Mar/20 16:02

Description

import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds

table = pa.table({'a': np.random.randn(10), 'b': range(10), 'c': ['a', 'b'] * 5})
pq.write_table(table, "test.parquet")

import pathlib

ds.dataset(pathlib.Path("./test.parquet"))
# gives UnionDataset

ds.dataset(str(pathlib.Path("./test.parquet")))
# correctly gives FileSystemDataset

and since those two dataset classes have different API, this is important to give FileSystemDataset

Issue Links

• links to: GitHub Pull Request #6783