[MAPREDUCE-1597] combinefileinputformat does not work with non-splittable files

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.22.0
• Hadoop Flags: Reviewed
• Assignee: Amareshwari Sriramadasu
• Reporter: Namit Jain
• Created: 12/Mar/10 21:57
• Updated: 27/Jul/15 11:39
• Resolved: 07/Sep/10 09:06

Description

CombineFileInputFormat.getSplits() does not take into account whether a file is splittable.
This can lead to a problem for compressed text files - for example, getSplits() may return more
than 1 split depending on the size of the compressed file, all the splits recordreader will read the
complete file.

I ran into this problem while using Hive on hadoop 20.

Issue Links

• blocks: HIVE-11376 CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are found for one of the input files
• duplicates: MAPREDUCE-1649 Compressed files with TextInputFormat does not work with CombineFileInputFormat
• is related to: SQOOP-721 Duplicating rows on export when exporting from compressed files.
• supercedes: HIVE-2089 Add a new input format to be able to combine multiple .gz text files