Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4790

Join after union fail due to UnionOptimizer

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.16.0
    • Component/s: tez
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The following script fail to run:

      rmf ooo
      
      a = load 'student.txt' as (name:chararray, age:int, gpa:double);
      b = filter a by age > 65;
      c = filter a by age <=10;
      d = union b, c;
      e = join a by name left, d by name;
      store e into 'ooo';
      

      Exception stack:

      Caused by: java.lang.IllegalArgumentException: Edge [scope-43 : org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor] -> [scope-55 : org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor] ({ SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager }) already defined!
              at org.apache.tez.dag.api.DAG.addEdge(DAG.java:272)
              at org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder.visitTezOp(TezDagBuilder.java:311)
              at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:252)
              at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56)
              at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:87)
              at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46)
              at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.buildDAG(TezJobCompiler.java:65)
              at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:111)
              ... 20 more
      

      Disable pig.tez.opt.union the script runs fine.

      Seems we shall detect this patten and disallow merge vertex group into a pair already has an edge.

        Attachments

        1. PIG-4790-3.patch
          18 kB
          Rohini Palaniswamy
        2. PIG-4790-2.patch
          12 kB
          Daniel Dai
        3. PIG-4790-1.patch
          12 kB
          Rohini Palaniswamy

          Activity

            People

            • Assignee:
              rohini Rohini Palaniswamy
              Reporter:
              daijy Daniel Dai
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: