[PIG-490] Combiner not used when group elements referred to in tuple notation instead of flatten.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.2.0
• Fix Version/s: 0.9.0
• Assignee: Thejas Nair
• Reporter: Alan Gates
• Created: 10/Oct/08 21:09
• Updated: 04/Aug/11 00:34
• Resolved: 19/Dec/10 04:00

Description

Given a query like:

A = load 'myfile';
B = group A by ($0, $1);
C = foreach B generate group.$0, group.$1, COUNT(A);

The combiner will not be invoked. But if the last line is changed to:

C = foreach B generate flatten(group), COUNT(A);

it will be. The reason for the discrepancy is because the CombinerOptimizer checks that all of the projections are simple. If not, it does not use the combiner. group.$0 is not a simple projection, so this is failed. However, this is a common enough case that the CombinerOptimizer should detect it and still use the combiner.

Issue Links

• is duplicated by: PIG-1297 algebraic interface of udf does not get used if the foreach with udf projects column within group