[ARROW-7673] [C++][Dataset] Revisit File discovery failure mode

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: C++
• Labels: dataset
• External issue URL: https://github.com/apache/arrow/issues/23918
• Assignee: Francois Saint-Jacques
• Reporter: Francois Saint-Jacques
• Created: 24/Jan/20 15:57
• Updated: 11/Jan/23 07:55
• Resolved: 08/Jun/20 15:56

Description

Currently, the default `FileSystemFactoryOptions::exclude_invalid_files` will silently ignore unsupported files (either IO error, not of the valid format, corruption, missing compression codecs, etc...) when creating a `FileSystemSource`.

We should change this behavior to propagate an error in the Inspect/Finish calls by default and allow the user to toggle `exclude_invalid_files`. The error should contain at least the file path and a decipherable error (if possible).

Issue Links

• is related to: ARROW-8283 [Python][Dataset] Non-existent files are silently dropped in pa.dataset.FileSystemDataset
• is related to: ARROW-8058 [C++][Python][Dataset] Provide an option to toggle validation and schema inference in FileSystemDatasetFactoryOptions