[HIVE-5834] Avoid reading ORC footers for files which will not be split in OrcInputFormat::getSplits()

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: tez-branch
• Fix Version/s: tez-branch
• Component/s: Tez
• Labels: perfomance split
• Release Note: Avoid reading ORC footers for files where data and footer are in the same HDFS block
• Assignee: Gopal Vijayaraghavan
• Reporter: Gopal Vijayaraghavan
• Created: 15/Nov/13 22:05
• Updated: 15/Nov/13 22:51
• Resolved: 15/Nov/13 22:51

Description

OrcInputFormat::getSplits() fires off a SplitGenerator task for every file in the task.

The footer & data are on the same node for all files with only 1 hdfs block. On top of that, it will never need a further split as long as its total size is < context.maxSize.

Reading that footer locally is faster than reading it in the split gen and sending it from the AM.