[HIVE-21924] Split text files even if header/footer exists

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0, 3.2.0, 4.0.0
• Fix Version/s: 4.0.0-alpha-1
• Component/s: File Formats
• Labels: pull-request-available
• Target Version/s: 3.2.0, 4.0.0
• Assignee: Mustafa İman
• Reporter: Prasanth Jayachandran
• Created: 26/Jun/19 01:33
• Updated: 17/Oct/23 13:10
• Resolved: 09/Oct/19 17:25

Description

https://github.com/apache/hive/blob/967a1cc98beede8e6568ce750ebeb6e0d048b8ea/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java#L494-L503

    int headerCount = 0;
    int footerCount = 0;
    if (table != null) {
      headerCount = Utilities.getHeaderCount(table);
      footerCount = Utilities.getFooterCount(table, conf);
      if (headerCount != 0 || footerCount != 0) {
        // Input file has header or footer, cannot be splitted.
        HiveConf.setLongVar(conf, ConfVars.MAPREDMINSPLITSIZE, Long.MAX_VALUE);
      }
    }

this piece of code makes the CSV (or any text files with header/footer) files not splittable if header or footer is present.
If only header is present, we can find the offset after first line break and use that to split. Similarly for footer, may be read few KB's of data at the end and find the last line break offset. Use that to determine the data range which can be used for splitting. Few reads during split generation are cheaper than not splitting the file at all.

Issue Links

• causes: HIVE-26284 ClassCastException: java.io.PushbackInputStream cannot be cast to org.apache.hadoop.fs.Seekable when table properties contains 'skip.header.line.count' = '1' and datafiles are in UTF-16 encoding
• contains: HIVE-21951 Llap query on external table with header or footer returns incorrect row count.
• Is contained by: HIVE-26751 Bug Fixes and Improvements for 3.2.0 release
• is fixed by: HIVE-24224 Fix skipping header/footer for Hive on Tez on compressed files
• is fixed by: HIVE-24381 Compressed text input returns 0 rows if skip header/footer is mentioned
• is fixed by: HIVE-22769 Incorrect query results and query failure during split generation for compressed text files
• links to: GitHub Pull Request #791
• links to: GitHub Pull Request #4728