[PIG-580] PERFORMANCE: Combiner should also be used when there are distinct aggregates in a foreach following a group provided there are no non-algebraics in the foreach

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.2.0
• Fix Version/s: 0.2.0
• Patch Info: Patch Available
• Assignee: Pradeep Kamath
• Reporter: Pradeep Kamath
• Created: 29/Dec/08 18:59
• Updated: 24/Mar/10 22:04
• Resolved: 05/Jan/09 23:58

Description

Currently Pig uses the combiner only when there is foreach following a group when the elements in the foreach generate have the following characteristics:
1) simple project of the "group" column
2) Algebraic UDF

The above conditions exclude use of the combiner for distinct aggregates - the distinct operation itself is combinable (irrespective of whether it feeds to an algebraic or non algebraic udf). So if the following foreach should also be combinable:

..
b = group a by $0;
c = foreach b generate { x = distinct a; generate group, COUNT(x), SUM(x.$1) }

The combiner optimizer should cause the distinct to be combined and the final combine output should feed the COUNT() and SUM() in the reduce.