[PIG-4843] Turn off combiner in reducer vertex for Tez if bags are in combine plan

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Hadoop Flags: Reviewed
• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 21/Mar/16 20:03
• Updated: 08/Jun/16 20:48
• Resolved: 22/Mar/16 10:40

Description

B = group A by key;
C = foreach B {
                                         key_value           =  A.key_value;
                                         distinct_key_value  = DISTINCT key_value;
                                         generate group, MIN(A.key_value) as min_value, MAX(A.key_value) as max_value, COUNT(distinct_key_value) as distinct_values;
                    }

In the above example, the combine plan holds the Distinct bag and it causes OOM when combiner is run by the MergeManager in the reducer. We did not have this issue with mapreduce as combiner is not running in reducer for new API till now (MAPREDUCE-5221)