Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5106

Refactor SkipRecordsInspector to exclude check for predefined file formats

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.9.0
    • Fix Version/s: 1.12.0
    • Component/s: Storage - Hive
    • Labels:
      None

      Description

      After changes introduced in DRILL-4982, SkipRecordInspector is used only for predefined formats (using hasHeaderFooter: false / true). But SkipRecordInspector has its own check for formats where skip strategy can be applied. Acceptable file formats are stored in private final Set<Object> fileFormats and initialized in constructor, currently it contains only one format - TextInputFormat. Now this check is redundant and may lead to ignoring hasHeaderFooter setting to true for any other format except of Text.

      To do:
      1. remove private final Set<Object> fileFormats
      2. remove if block from SkipRecordsInspector.retrievePositiveIntProperty:

       if (!fileFormats.contains(tableProperties.get(hive_metastoreConstants.FILE_INPUT_FORMAT))) {
      return propertyIntValue;
      }
      

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              arina Arina Ielchiieva
              Reporter:
              arina Arina Ielchiieva

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment