[DRILL-5106] Refactor SkipRecordsInspector to exclude check for predefined file formats

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.12.0
• Component/s: Storage - Hive
• Assignee: Arina Ielchiieva
• Reporter: Arina Ielchiieva
• Created: 05/Dec/16 12:18
• Updated: 23/Nov/17 13:28
• Resolved: 23/Nov/17 13:28

Description

After changes introduced in DRILL-4982, SkipRecordInspector is used only for predefined formats (using hasHeaderFooter: false / true). But SkipRecordInspector has its own check for formats where skip strategy can be applied. Acceptable file formats are stored in private final Set<Object> fileFormats and initialized in constructor, currently it contains only one format - TextInputFormat. Now this check is redundant and may lead to ignoring hasHeaderFooter setting to true for any other format except of Text.

To do:
1. remove private final Set<Object> fileFormats
2. remove if block from SkipRecordsInspector.retrievePositiveIntProperty:

 if (!fileFormats.contains(tableProperties.get(hive_metastoreConstants.FILE_INPUT_FORMAT))) {
return propertyIntValue;
}

Issue Links

• is part of: DRILL-5941 Skip header / footer logic works incorrectly for Hive tables when file has several input splits
• relates to: DRILL-4982 Hive Queries degrade when queries switch between different formats