[ARROW-9644] [C++][Dataset] Do not check for ignore_prefixes in the base path

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.1, 2.0.0
• Component/s: C++, Python, R
• Labels: dataset pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25704
• Assignee: Ben Kietzman
• Reporter: Joris Van den Bossche
• Created: 04/Aug/20 08:04
• Updated: 11/Jan/23 08:08
• Resolved: 13/Aug/20 14:19

Description

Somewhat related to ARROW-8427, and from https://github.com/apache/arrow/issues/7857

I am not sure we should check the ignore_prefixes for the base path provided by the user. Because if that contains eg an underscore, it simply skips the full dataset resulting in an empty dataset.

import tempfile
import pathlib

path = tempfile.mkdtemp()
tmpdir =  pathlib.Path(path)                                                                                                                                                              

# base path with a directory with an underscore 
datadir = tmpdir / "_data" / "dataset"                                                                                                                                                                    
datadir.mkdir(parents=True, exist_ok=True)                                                                                                                                                                

# create a parquet file at that location
import pyarrow as pa
import pyarrow.parquet as pq

table = pa.table({'a': [1, 2, 3]})                                                                                                                                                                        
pq.write_table(table, datadir / "data.parquet")                                                                                                                                                           

# reading dataset skips everything
import pyarrow.dataset as ds                                                                                                                                                                              

In [26]: ds.dataset(datadir)                                                                                                                                                                                       
Out[26]: <pyarrow._dataset.FileSystemDataset at 0x7fbfd8779bb0>

In [27]: ds.dataset(datadir).files                                                                                                                                                                                 
Out[27]: []

cc bkietz npr

Issue Links

• is duplicated by: ARROW-9675 the table does not load if the file path has a dot at start
• links to: GitHub Pull Request #7907