[ARROW-9748] [C++][Dataset] Remove Selector, ignore_prefixes from FileSystemDatasetFactory

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.0.0
• Component/s: C++
• Labels: dataset
• External issue URL: https://github.com/apache/arrow/issues/25800
• Assignee: Unassigned
• Reporter: Ben Kietzman
• Created: 14/Aug/20 20:16
• Updated: 11/Jan/23 08:08

Description

Currently FileSystemDatasetFactory can be constructed with an explicit listing of files or with a fs::FileSelector. Since the selector does not support sophisticated selection criteria, FileSystemFactoryOptions::selector_ignore_prefixes to allow users to exclude undesired files such as _metadata or .DS_STORE.

The selector + ignored prefixes mechanism is inflexible with numerous edge cases ( ARROW-9644 ARROW-9573 ). Furthermore, implementing more advanced file selection logic in dataset discovery prevents it from being reused by other consumers of the file system api.

Remove FileSystemDatasetFactory's constructor-from-selector, optionally adding that functionality directly to fs::FileSelector. An explicit listing of files for use in construction of a FileSystemDatasetFactory can then be assembled using an fs::FileSelector and/or other globbing libraries, with arbitrary inclusion logic.

Issue Links

• relates to: ARROW-9657 [R][Dataset] Expose more FileSystemDatasetFactory options