[PIG-5082] Tez UnionOptimizer creates vertex group with one member

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.16.0
• Fix Version/s: 0.17.0, 0.16.1
• Component/s: tez
• Assignee: Rohini Palaniswamy
• Reporter: Travis Woodruff
• Created: 21/Dec/16 22:36
• Updated: 21/Jun/17 09:15
• Resolved: 17/Jan/17 15:47

Description

This script results in a vertex group with one member:

a = LOAD '/tmp/empty.txt' USING PigStorage('\t') AS (x:chararray);
b = LOAD '/tmp/empty.txt' USING PigStorage('\t') AS (x:chararray);
c = LOAD '/tmp/empty.txt' USING PigStorage('\t') AS (y:chararray);

u1 = UNION ONSCHEMA a, b;
SPLIT u1 INTO r IF x != '', s OTHERWISE;
d = JOIN r BY x LEFT, c BY y;
u2 = UNION ONSCHEMA d, s;
e = FILTER u2 BY x == '';
f = FILTER u2 BY x == 'm';
u3 = UNION ONSCHEMA e, f;
DUMP u3;

Which results in:

java.lang.IllegalArgumentException: VertexGroup must have at least 2 members
	at org.apache.tez.dag.api.VertexGroup.<init>(VertexGroup.java:77)
	at org.apache.tez.dag.api.DAG.createVertexGroup(DAG.java:202)
	at org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder.visitTezOp(TezDagBuilder.java:396)
	at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:255)
...

This seems to be happening because UnionOptimizer is replacing a union with a vertex group and then optimizing away a predecessor union thus removing a node and resulting in a vertex group with one member.

Issue Links

• is part of: PIG-5078 Script fails with error - POStoreTez only accepts MROutput