[HIVE-1357] CombineHiveInputSplit should initialize the inputFileFormat once for a single split

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Component/s: Query Processor, Serializers/Deserializers
• Hadoop Flags: Reviewed
• Assignee: Ning Zhang
• Reporter: Ning Zhang
• Created: 21/May/10 01:04
• Updated: 17/Dec/11 00:03
• Resolved: 21/May/10 18:08

Description

If a split consists of multiple files, the FileFormat should always be the same, whether RCFile or SequenceFile. Currently the CombineHiveInputSplit tries to get the inputFileFormat for each new file in the split, which is O where n is the number of files in the split. This is an O(n^2) operation and degrade the performance badly for combining large number of small files.