[ARROW-8283] [Python][Dataset] Non-existent files are silently dropped in pa.dataset.FileSystemDataset

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Python
• Labels: dataset pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/24477
• Assignee: Joris Van den Bossche
• Reporter: Uwe Korn
• Created: 31/Mar/20 12:40
• Updated: 11/Jan/23 07:59
• Resolved: 18/Jun/20 15:22

Description

When passing a list of files to the constructor of pyarrow.dataset.FileSystemDataset, all files that don't exist are silently dropped immediately (i.e. no fragments are created for them).

Instead, I would expect that fragments will be created for them but an error is thrown when one tries to read the fragment with the non-existent file.

Issue Links

• relates to: ARROW-7673 [C++][Dataset] Revisit File discovery failure mode
• links to: GitHub Pull Request #7468