[PIG-750] Use combiner when algebraic UDFs are used in expressions

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Hadoop Flags: Reviewed

Release Note

With changes in the patch, queries which have algebraic functions within expressions also will use combiner. This is as long as the bags from group-by are only input for algebraic expressions. If bag is projected or a non algebraic expression/udf has bag as input, combiner will not be used.
Combiner will be used in case of following foreach statements (that follow group) -
describe B ;
B: {group: int, A: {c1 : int, c2 : int, c3 : int}}

1) foreach B generate SUM(A.c2) * AVG(A.c3), ...
2) foreach B generate 1 / SUM(A.c2)
3) foreach B generate EXP(AVG(A.c2))
4) foreach B generate group + SUM(A.c2)


Following statements will not use combiner -
1) foreach B generate A.c2, ...
2) foreach B generate EXP(c2) , SUM(c2) ... - Where EXP is non algebraic function

In case of nested foreach statement, if it has limit, order, or filter , combiner does not get used (as before).

This patch also fixes PIG-490, foreach statements that access group elements also use combiner
for example -
1) foreach B generate group.$0, group.$1, COUNT(A);
1) foreach B generate group.c1, group.c2, COUNT(A);

• Assignee: Thejas Nair
• Reporter: Amir Youssefi
• Created: 02/Apr/09 23:59
• Updated: 04/Aug/11 00:34
• Resolved: 19/Dec/10 03:59

Description

Currently Pig uses combiner when all a,b, c,... are algebraic (e.g. SUM, AVG etc.) in foreach:

foreach X generate a,b,c,...

It's a performance improvement if it uses combiner when a mix of algebraic and non-algebraic functions are used as well.