[PIG-5444] TestFRJoin.testFRJoinOut7 and testFRJoinOut8 failing with Edge already defined error on Tez

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: tez
• Assignee: Koji Noguchi
• Reporter: Koji Noguchi
• Created: 12/Nov/23 06:38
• Updated: 25/Sep/24 19:25

Description

With Tez, when testing individual tests (TestFRJoin.testFRJoinOut7 and testFRJoinOut8) separately, they pass the tests. But when entire TestFRJoin is run, these two tests on Tez are failing with

Unable to open iterator for alias E
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias E
at org.apache.pig.PigServer.openIterator(PigServer.java:1024)
at org.apache.pig.test.TestFRJoin.testFRJoinOut7(TestFRJoin.java:409)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias E
at org.apache.pig.PigServer.storeEx(PigServer.java:1127)
at org.apache.pig.PigServer.store(PigServer.java:1086)
at org.apache.pig.PigServer.openIterator(PigServer.java:999)
Caused by: org.apache.pig.backend.hadoop.executionengine.JobCreationException: ERROR 2017: Internal error creating job configuration.
at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:153)
at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.compile(TezJobCompiler.java:81)
at org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:200)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1479)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464)
at org.apache.pig.PigServer.storeEx(PigServer.java:1123)
Caused by: java.lang.IllegalArgumentException: Edge [scope-632 : org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor] -> [scope-628 : org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor] ({ BROADCAST : org.apache.tez.runtime.library.input.UnorderedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.UnorderedKVOutput >> NullEdgeManager }) already defined!
at org.apache.tez.dag.api.DAG.addEdge(DAG.java:296)
at org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder.visitTezOp(TezDagBuilder.java:410)
at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:265)
at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56)
at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:87)
at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46)
at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.buildDAG(TezJobCompiler.java:69)
at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:120)