Major Description
We are trying to enable VolcanoPlanner in Hive for some specific cases. Doing experiments, I tried to set up the impatient option to true. Then I am hitting an assertion error in ChunkList with the following stack trace:
select cbo_t3.c_int, c, count(*) from (select key as a, c_int+1 as b, sum(c_int) as c from cbo_t1 where (cbo_t1.c_int + 1 >= 0) and (cbo_t1.c_int > 0 or cbo_t1.c_float >= 0) group by c_float, cbo_t1.c_int, key order by a) cbo_t1 join (select key as p, c_int+1 as q, sum(c_int) as r from cbo_t2 where (cbo_t2.c_int + 1 >= 0) and (cbo_t2.c_int > 0 or cbo_t2.c_float >= 0) group by c_float, cbo_t2.c_int, key order by q/10 desc, r asc) cbo_t2 on cbo_t1.a=p join cbo_t3 on cbo_t1.a=key where (b + cbo_t2.q >= 0) and (b > 0 or c_int >= 0) group by cbo_t3.c_int, c order by cbo_t3.c_int+c desc, c See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log, or check ./ql/target/surefire-reports or ./itests/qtest/target/surefire-reports/ for specific test cases logs. TestCliDriver.testCliDriver_correlationoptimizer8:108->runTest:134 Unexpected exception java.lang.AssertionError at org.apache.calcite.util.ChunkList$ChunkListIterator.next(ChunkList.java:217) at java.util.AbstractList.removeRange(AbstractList.java:570) at java.util.AbstractList.clear(AbstractList.java:234) at org.apache.calcite.plan.volcano.RuleQueue$PhaseMatchList.clear(RuleQueue.java:714) at org.apache.calcite.plan.volcano.RuleQueue.phaseCompleted(RuleQueue.java:155) at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:833) at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.volcanoPlan(CalcitePlanner.java:1244) at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:1090) at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:887) at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:830) at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:112) at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:971) at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:148) at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:105) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:653) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:270) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10094) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:229) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237) at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:471) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1149) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1237) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1075) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1063) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:400) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336) at org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1127) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1101) at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8(TestCliDriver.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
It seems the problem originates when we call the clean method in the ChunkList.