[PIG-2009] Better MergeForEach rule

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.0
• Assignee: Daniel Dai
• Reporter: Daniel Dai
• Created: 21/Apr/11 23:30
• Updated: 04/Oct/11 20:09

Description

MergeForEach rule will not merge two consecutive ForEach if the second ForEach has inner relational plan. This prevent some optimizations. Eg,

A = LOAD 'input1' AS (a0, a1, a2);
B = LOAD 'input2' AS (b0, b1, b2);
C = cogroup A by a0, B by b0;
D = foreach C { E = limit A 10; F = E.a1; G = DISTINCT F; generate group, COUNT(G);};
explain D;

We add ForEach after cogroup to prune B, however, we cannot merge this ForEach with D. Secondary key optimization for this query is thus disabled.