[HIVE-22672] Error in tez optimizing SQL statement

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.2.0
• Component/s: hpl/sql
• Assignee: Unassigned
• Reporter: yinghua_zh
• Created: 27/Dec/19 00:51
• Updated: 27/Dec/19 00:51

Description

(): FAILED: ConcurrentModificationException null
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
at java.util.HashMap$KeyIterator.next(HashMap.java:1461)
at org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationFromSMBJoins(TezCompiler.java:640)
at org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:117)
at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:138)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11169)
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:257)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:485)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1252)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1239)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:191)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:276)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:312)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:500)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:487)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)

 

When map traverses, its elements are deleted，The code as follow:

for (TableScanOperator ts : tsOps) {
for (ReduceSinkOperator rs : pctx.getRsOpToTsOpMap().keySet()) {
if (ts == pctx.getRsOpToTsOpMap().get(rs))

{ // match! GenTezUtils.removeBranch(rs); GenTezUtils.removeSemiJoinOperator(pctx, rs, ts); }

}
}

The  function:GenTezUtils.removeSemiJoinOperator(pctx, rs, ts) will delete the eletement from pctx.getRsOpToTsOpMap().