[MAPREDUCE-1248] Redundant memory copying in StreamKeyValUtil

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.22.0
• Fix Version/s: 0.22.0
• Component/s: contrib/streaming
• Hadoop Flags: Reviewed
• Assignee: Ruibang He
• Reporter: Ruibang He
• Created: 30/Nov/09 09:47
• Updated: 23/Nov/11 06:12
• Resolved: 08/Jul/10 06:04

Description

I found that when MROutputThread collecting the output of Reducer, it calls StreamKeyValUtil.splitKeyVal() and two local byte-arrays are allocated there for each line of output. Later these two byte-arrays are passed to variable key and val. There are twice memory copying here, one is the System.arraycopy() method, the other is inside key.set() / val.set().

This causes double times of memory copying for the whole output (may lead to higher CPU consumption), and frequent temporay object allocation.