[ARROW-7061] [C++][Dataset] FileSystemDiscovery with ParquetFileFormat should ignore files that aren't Parquet

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Component/s: C++
• Labels: dataset pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/23370
• Assignee: Francois Saint-Jacques
• Reporter: Neal Richardson
• Created: 04/Nov/19 23:11
• Updated: 11/Jan/23 07:51
• Resolved: 15/Nov/19 20:08

Description

I got Invalid parquet file. Corrupt footer. trying to read real data. Turned out it was because I had opened the directory in macOS Finder and it had added the junk .DS_Store files. Once I deleted them, the Dataset created fine.

If we're creating a DataSource with Parquet files, we should ignore any non-Parquet files we encounter when scanning.

Issue Links

• links to: GitHub Pull Request #5811