Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-5082

Tez UnionOptimizer creates vertex group with one member

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.16.0
    • Fix Version/s: 0.17.0, 0.16.1
    • Component/s: tez
    • Labels:
      None

      Description

      This script results in a vertex group with one member:

      a = LOAD '/tmp/empty.txt' USING PigStorage('\t') AS (x:chararray);
      b = LOAD '/tmp/empty.txt' USING PigStorage('\t') AS (x:chararray);
      c = LOAD '/tmp/empty.txt' USING PigStorage('\t') AS (y:chararray);
      
      u1 = UNION ONSCHEMA a, b;
      SPLIT u1 INTO r IF x != '', s OTHERWISE;
      d = JOIN r BY x LEFT, c BY y;
      u2 = UNION ONSCHEMA d, s;
      e = FILTER u2 BY x == '';
      f = FILTER u2 BY x == 'm';
      u3 = UNION ONSCHEMA e, f;
      DUMP u3;
      

      Which results in:

      java.lang.IllegalArgumentException: VertexGroup must have at least 2 members
      	at org.apache.tez.dag.api.VertexGroup.<init>(VertexGroup.java:77)
      	at org.apache.tez.dag.api.DAG.createVertexGroup(DAG.java:202)
      	at org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder.visitTezOp(TezDagBuilder.java:396)
      	at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:255)
      ...
      

      This seems to be happening because UnionOptimizer is replacing a union with a vertex group and then optimizing away a predecessor union thus removing a node and resulting in a vertex group with one member.

        Attachments

        1. PIG-5082.patch
          2 kB
          Travis Woodruff

          Issue Links

            Activity

              People

              • Assignee:
                rohini Rohini Palaniswamy
                Reporter:
                tmwoodruff Travis Woodruff
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: