[ARROW-8058] [C++][Python][Dataset] Provide an option to toggle validation and schema inference in FileSystemDatasetFactoryOptions

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.16.0
• Fix Version/s: 0.17.0
• Component/s: C++, Python
• Labels: dataset pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/24271
• Assignee: Francois Saint-Jacques
• Reporter: Ben Kietzman
• Created: 10/Mar/20 13:21
• Updated: 11/Jan/23 07:57
• Resolved: 25/Mar/20 17:01

Description

This can be costly and is not always necessary.

At the same time we could move file validation into the scan tasks; currently all files are inspected as the dataset is constructed, which can be expensive if the filesystem is slow. We'll be performing the validation multiple times but the check will be cheap since at scan time we'll be reading the file into memory anyway.

Issue Links

• relates to: ARROW-7673 [C++][Dataset] Revisit File discovery failure mode
• links to: GitHub Pull Request #6687