Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: core
    • Labels:
      None

      Description

      We are trying to enable VolcanoPlanner in Hive for some specific cases. Doing experiments, I tried to set up the impatient option to true. Then I am hitting an assertion error in ChunkList with the following stack trace:

      select cbo_t3.c_int, c, count(*) from (select key as a, c_int+1 as b, sum(c_int) as c from cbo_t1 where (cbo_t1.c_int + 1 >= 0) and (cbo_t1.c_int > 0 or cbo_t1.c_float >= 0) group by c_float, cbo_t1.c_int, key order by a) cbo_t1 join (select key as p, c_int+1 as q, sum(c_int) as r from cbo_t2 where (cbo_t2.c_int + 1 >= 0) and (cbo_t2.c_int > 0 or cbo_t2.c_float >= 0)  group by c_float, cbo_t2.c_int, key order by q/10 desc, r asc) cbo_t2 on cbo_t1.a=p join cbo_t3 on cbo_t1.a=key where (b + cbo_t2.q >= 0) and (b > 0 or c_int >= 0) group by cbo_t3.c_int, c order by cbo_t3.c_int+c desc, c
      See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log, or check ./ql/target/surefire-reports or ./itests/qtest/target/surefire-reports/ for specific test cases logs.
        TestCliDriver.testCliDriver_correlationoptimizer8:108->runTest:134 Unexpected exception java.lang.AssertionError
      	at org.apache.calcite.util.ChunkList$ChunkListIterator.next(ChunkList.java:217)
      	at java.util.AbstractList.removeRange(AbstractList.java:570)
      	at java.util.AbstractList.clear(AbstractList.java:234)
      	at org.apache.calcite.plan.volcano.RuleQueue$PhaseMatchList.clear(RuleQueue.java:714)
      	at org.apache.calcite.plan.volcano.RuleQueue.phaseCompleted(RuleQueue.java:155)
      	at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:833)
      	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.volcanoPlan(CalcitePlanner.java:1244)
      	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:1090)
      	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:887)
      	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:830)
      	at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:112)
      	at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:971)
      	at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:148)
      	at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:105)
      	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:653)
      	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:270)
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10094)
      	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:229)
      	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
      	at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
      	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
      	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:471)
      	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
      	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1149)
      	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1237)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1075)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1063)
      	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
      	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:400)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
      	at org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1127)
      	at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1101)
      	at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124)
      	at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8(TestCliDriver.java:108)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      

      It seems the problem originates when we call the clean method in the ChunkList.

        Activity

        Hide
        julianhyde Julian Hyde added a comment -

        Resolved in release 1.6.0 (2016-01-22).

        Show
        julianhyde Julian Hyde added a comment - Resolved in release 1.6.0 (2016-01-22).
        Show
        julianhyde Julian Hyde added a comment - Fixed in http://git-wip-us.apache.org/repos/asf/calcite/commit/3cba7055 .
        Hide
        julianhyde Julian Hyde added a comment - - edited

        I have reproduced this. The problem occurs when you invoke clear on a chunk list of size 2 or more. It is something to do with positioning of iterators after remove has been called.

        Clearly we can provide an optimized version of clear rather than relying on repeated remove. But we should also fix the iterator.

        Work in progress at https://github.com/julianhyde/calcite/tree/1047-chunk-list.

        Show
        julianhyde Julian Hyde added a comment - - edited I have reproduced this. The problem occurs when you invoke clear on a chunk list of size 2 or more. It is something to do with positioning of iterators after remove has been called. Clearly we can provide an optimized version of clear rather than relying on repeated remove . But we should also fix the iterator. Work in progress at https://github.com/julianhyde/calcite/tree/1047-chunk-list .

          People

          • Assignee:
            julianhyde Julian Hyde
            Reporter:
            jcamachorodriguez Jesus Camacho Rodriguez
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development