Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5106

Refactor SkipRecordsInspector to exclude check for predefined file formats

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.9.0
    • Fix Version/s: 1.12.0
    • Component/s: Storage - Hive
    • Labels:
      None

      Description

      After changes introduced in DRILL-4982, SkipRecordInspector is used only for predefined formats (using hasHeaderFooter: false / true). But SkipRecordInspector has its own check for formats where skip strategy can be applied. Acceptable file formats are stored in private final Set<Object> fileFormats and initialized in constructor, currently it contains only one format - TextInputFormat. Now this check is redundant and may lead to ignoring hasHeaderFooter setting to true for any other format except of Text.

      To do:
      1. remove private final Set<Object> fileFormats
      2. remove if block from SkipRecordsInspector.retrievePositiveIntProperty:

       if (!fileFormats.contains(tableProperties.get(hive_metastoreConstants.FILE_INPUT_FORMAT))) {
      return propertyIntValue;
      }
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                arina Arina Ielchiieva
                Reporter:
                arina Arina Ielchiieva
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: