[DRILL-2577] Parquet scan fails when directory contains _SUCCESS or _logs

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.9.0
• Component/s: Storage - Parquet
• Assignee: Steven Phillips
• Reporter: Steven Phillips
• Created: 26/Mar/15 01:34
• Updated: 15/Apr/15 08:57
• Resolved: 15/Apr/15 08:57

Description

_SUCCESS and _logs are often created from map reduce jobs, and typically ignored. This is commonly done using the OutputFilesFilter in hadoop.
The new FooterGatherer class, which is used to read parquet footers in parallel, does not use this filter. So it attempts to read these as parquet files and fails.
The fix is to use the DrillPathFilter, which extends OutputFilesFilter, in the FooterGather code.

Issue Links

• relates to: DRILL-1131 Drill should ignore files in starting with . _