[HIVE-2350] Improve RCFile Read Speed

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Component/s: Serializers/Deserializers
• Labels: performance rcfile
• Hadoop Flags: Reviewed
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 05/Aug/11 03:01
• Updated: 16/Dec/11 23:56
• Resolved: 23/Aug/11 02:26

Description

By tweaking the RCFile$Reader implementation to allow more efficient memory access I was able to reduce CPU usage. I measured the speed required to scan a gzipped RCFile, decompress and assemble into records. CPU time was reduced by about 7% for a full table scan, An improvement of about 2% was realised when a smaller subset of columns (3-5 out of tens) were selected.