Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-5375

NullPointerException for multi-level self unions with Tez UnionOptimizer

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      A = load 'input.txt' as (a0:int, a1: chararray, a2:int);
      B = load 'input.txt' as (a0:int, a1: chararray, a2:int);
      C = load 'input.txt' as (a0:int, a1: chararray, a2:int);
      
      A_and_B = UNION A, B;
      SPLIT A_and_B INTO A_and_B2 IF a0 > 10, A_and_B3 OTHERWISE;
      
      A_and_B_and_C = UNION ONSCHEMA C, A_and_B;
      
      X = UNION ONSCHEMA A_and_B_and_C, A_and_B2, A_and_B3;
      
      X2 = GROUP X ALL ;
      
      dump X2;
      

      This fails on Tez with

      Pig Stack Trace
      ---------------
      ERROR 1002: Unable to store alias X2
      
      org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias X2
              at org.apache.pig.PigServer.openIterator(PigServer.java:1024)
              at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:790)
              at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:383)
              at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231)
              at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206)
              at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
              at org.apache.pig.Main.run(Main.java:630)
              at org.apache.pig.Main.main(Main.java:175)
      Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias X2
              at org.apache.pig.PigServer.storeEx(PigServer.java:1127)
              at org.apache.pig.PigServer.store(PigServer.java:1086)
              at org.apache.pig.PigServer.openIterator(PigServer.java:999)
              ... 7 more
      Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: java.lang.NullPointerException
              at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:296)
              at org.apache.pig.PigServer.launchPlan(PigServer.java:1479)
              at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464)
              at org.apache.pig.PigServer.storeEx(PigServer.java:1123)
              ... 9 more
      Caused by: java.lang.NullPointerException
              at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.patchPackage(TezPOPackageAnnotator.java:97)
              at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.handlePackage(TezPOPackageAnnotator.java:78)
              at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.visitTezOp(TezPOPackageAnnotator.java:61)
              at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:265)
              at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56)
              at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
              at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71)
              at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71)
              at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:52)
              at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46)
              at org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:197)
              at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290)
              ... 12 more
      

        Attachments

        1. pig-5375-v1.patch
          19 kB
          Koji Noguchi
        2. pig-5375.png
          235 kB
          Koji Noguchi

          Activity

            People

            • Assignee:
              knoguchi Koji Noguchi
              Reporter:
              knoguchi Koji Noguchi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: