[TEZ-1003] Need a input that merges multiple ShuffleMergedInput from VertexGroup

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 31/Mar/14 09:24
• Updated: 06/Sep/14 01:35
• Resolved: 22/Apr/14 06:38

Description

In PIG-3835, was trying to do use vertex groups for unions. Union followed by store works fine. But when trying to do groupby,

A = LOAD '/tmp/data' AS (f1:int,f2:int);
B = LOAD '/tmp/data2' AS (f1:int,f2:int);
C = UNION onschema A,B;
D = GROUP C by f1;
E = FOREACH D GENERATE group, SUM(C.f2);
store E into '/tmp/tezout' using PigStorage();

ConcatenatedMergedKeyValuesInput on the reduce, had only grouped records within each input and not across all inputs.

i.e If A had records
a 1
b 1
b 2
and B
a 2
a 3
b 3

The records from ConcatenatedMergedKeyValuesInput of A and B were
a

{1}

, b

{1,2}

, a

{2,3}

, b

{3}

while I am expecting a

{1,2,3}, b {1,2,3}

Issue Links

• is depended upon by: PIG-3855 Turn on UnionOptimizer by default and add new e2e tests for union
• is required by: PIG-3835 Improve performance of union
• relates to: TEZ-1052 Optimize implementation of CompositeShuffledMergedInput