[CHUKWA-647] Spread out intermediate data with the same ReduceType into different Reduce Tasks

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.4.0, 0.6.0
• Fix Version/s: 0.6.0
• Component/s: Data Processors
• Assignee: Ariel Shemaiah Rabkin
• Reporter: Jie Huang
• Created: 16/Jul/12 05:28
• Updated: 17/Jul/12 00:48
• Resolved: 16/Jul/12 23:51

Description

We have found that if we partitioned the map output data according to ReduceType, we can see the data skew in some HiTune cases. Then one or two Reduce Tasks slow down the whole Demux job somehow, since those reduce tasks have to process more input-data.