[PIG-3826] Outer join with PushDownForEachFlatten generates wrong result

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.12.1, 0.13.0
• Component/s: impl
• Hadoop Flags: Reviewed
• Assignee: Daniel Dai
• Reporter: Daniel Dai
• Created: 21/Mar/14 02:12
• Updated: 15/Apr/14 20:45
• Resolved: 21/Mar/14 03:58

Description

The following script generates wrong result:
A = load 'A.txt' using PigStorage(',') as (id:chararray, value:double);
B = load 'B.txt' using PigStorage(',') as (id:chararray, name:chararray);

t1 = group A by id;
t2 = foreach t1

{ r1 = filter $1 by (value>1); r2 = limit r1 1; generate group as id, FLATTEN(r2.value) as value; }

;

t3 = join B by id LEFT OUTER, t2 by id;
dump t3;

A.txt:
1,1.5
2,0
3,-2.0
4,8.9

B.txt:
1,Ofer
2,Jordan
3,Noa
4,Daniel

Expected output:
(1,Ofer,1,1.5)
(2,Jordan,,)
(3,Noa,,)
(4,Daniel,4,8.9)

But we get:
(1,Ofer,1,1.5)
(4,Daniel,4,8.9)

With the option "-t PushDownForEachFlatten", the issue goes away.