Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-5444

TestFRJoin.testFRJoinOut7 and testFRJoinOut8 failing with Edge already defined error on Tez

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • tez
    • None

    Description

      With Tez, when testing individual tests (TestFRJoin.testFRJoinOut7 and testFRJoinOut8) separately, they pass the tests. But when entire TestFRJoin is run, these two tests on Tez are failing with

      Unable to open iterator for alias E
      org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias E
      at org.apache.pig.PigServer.openIterator(PigServer.java:1024)
      at org.apache.pig.test.TestFRJoin.testFRJoinOut7(TestFRJoin.java:409)
      Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias E
      at org.apache.pig.PigServer.storeEx(PigServer.java:1127)
      at org.apache.pig.PigServer.store(PigServer.java:1086)
      at org.apache.pig.PigServer.openIterator(PigServer.java:999)
      Caused by: org.apache.pig.backend.hadoop.executionengine.JobCreationException: ERROR 2017: Internal error creating job configuration.
      at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:153)
      at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.compile(TezJobCompiler.java:81)
      at org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:200)
      at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290)
      at org.apache.pig.PigServer.launchPlan(PigServer.java:1479)
      at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464)
      at org.apache.pig.PigServer.storeEx(PigServer.java:1123)
      Caused by: java.lang.IllegalArgumentException: Edge [scope-632 : org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor] -> [scope-628 : org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor] ({ BROADCAST : org.apache.tez.runtime.library.input.UnorderedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.UnorderedKVOutput >> NullEdgeManager }) already defined!
      at org.apache.tez.dag.api.DAG.addEdge(DAG.java:296)
      at org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder.visitTezOp(TezDagBuilder.java:410)
      at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:265)
      at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56)
      at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:87)
      at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46)
      at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.buildDAG(TezJobCompiler.java:69)
      at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:120)
      

      Attachments

        1. pig-5444-v03.patch
          7 kB
          Koji Noguchi
        2. pig-5444-v02.patch
          7 kB
          Koji Noguchi

        Activity

          People

            knoguchi Koji Noguchi
            knoguchi Koji Noguchi
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: