Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0-incubating
    • Component/s: None
    • Labels:
      None

      Description

      If AggregateJoinTransposeRule and AggregateProjectMergeRule are both enabled, and you run a query with grouping sets, you can get an AssertionError. For example,

      Error while executing command CheckResultCommand [sql: select emp.deptno as e, dept.deptno as d
      > from "scott".emp join "scott".dept using (deptno)
      > group by cube(emp.deptno, dept.deptno)
      > ]
      > java.lang.AssertionError: Internal error: Error while applying rule AggregateProjectMergeRule, args [rel#1144:LogicalAggregate.NONE.[](input=rel#1143:Subset#4.NONE.[],group={0, 1},groups=[{0, 1}, {0}, {1}, {}],indicator=true), rel#1163:LogicalProject.NONE.[](input=rel#1162:Subset#7.NONE.[],DEPTNO=$1,DEPTNO0=$0)]
      > 	at org.apache.calcite.util.Util.newInternal(Util.java:743)
      > 	at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
      > 	at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:798)
      > 	at org.apache.calcite.tools.Programs$5.run(Programs.java:272)
      > 	at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:320)
      > 	at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:142)
      > 	at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:280)
      > 	at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:188)
      > 	at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:521)
      > 	at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:417)
      > 	at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:386)
      > 	at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:174)
      > 	at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:489)
      > 	at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:474)
      > 	at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:109)
      > 	at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:129)
      > 	at net.hydromatic.quidem.Quidem$CheckResultCommand.execute(Quidem.java:690)
      > 	at net.hydromatic.quidem.Quidem$CompositeCommand.execute(Quidem.java:926)
      > 	at net.hydromatic.quidem.Quidem.execute(Quidem.java:193)
      > 	at org.apache.calcite.test.JdbcTest.checkRun(JdbcTest.java:4401)
      > 	at org.apache.calcite.test.JdbcTest.testRunJoin(JdbcTest.java:4349)
      > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      > 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      > 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      > 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      > 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      > 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
      > 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
      > 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
      > 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      > 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      > 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      > 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      > 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      > 	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      > 	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
      > 	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:78)
      > 	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:212)
      > 	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:68)
      > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      > 	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
      > Caused by: java.lang.AssertionError: [{0, 1}, {1}, {0}, {}]
      > 	at org.apache.calcite.rel.core.Aggregate.<init>(Aggregate.java:132)
      > 	at org.apache.calcite.rel.logical.LogicalAggregate.<init>(LogicalAggregate.java:65)
      > 	at org.apache.calcite.rel.logical.LogicalAggregate.copy(LogicalAggregate.java:105)
      > 	at org.apache.calcite.rel.logical.LogicalAggregate.copy(LogicalAggregate.java:43)
      > 	at org.apache.calcite.rel.rules.AggregateProjectMergeRule.apply(AggregateProjectMergeRule.java:114)
      > 	at org.apache.calcite.rel.rules.AggregateProjectMergeRule.onMatch(AggregateProjectMergeRule.java:61)
      > 	at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)

      The problem is that if you have a sorted list of grouping sets [[0, 2], [0], [2], []] and you permute the fields 0->2 and 2->1, then the permuted result [[2, 1], [2], [1], []] is no longer sorted. ([2] should occur after [1] but occurs before.)

        Issue Links

          Activity

          Hide
          julianhyde Julian Hyde added a comment -

          I discovered a workaround: don’t enable both AggregateJoinTransposeRule and AggregateProjectMergeRule. Since AggregateJoinTransposeRule is a new rule and AggregateProjectMergeRule is not part of Calcite’s standard rule set, I don’t believe that people will hit this issue.

          Show
          julianhyde Julian Hyde added a comment - I discovered a workaround: don’t enable both AggregateJoinTransposeRule and AggregateProjectMergeRule. Since AggregateJoinTransposeRule is a new rule and AggregateProjectMergeRule is not part of Calcite’s standard rule set, I don’t believe that people will hit this issue.
          Hide
          julianhyde Julian Hyde added a comment -

          Fixed in my personal github repo, https://github.com/julianhyde/incubator-calcite/commit/569aedf8aa6f2885bbe7cea25de503f57f472bed, and equivalently in the attached patch. I'll mark fixed when the patch is checked into master.

          Show
          julianhyde Julian Hyde added a comment - Fixed in my personal github repo, https://github.com/julianhyde/incubator-calcite/commit/569aedf8aa6f2885bbe7cea25de503f57f472bed , and equivalently in the attached patch. I'll mark fixed when the patch is checked into master.
          Hide
          jcamachorodriguez Jesus Camacho Rodriguez added a comment -

          LGTM, including the new methods that were added to ImmutableBitSet.

          Julian Hyde, for sake of clarity, although this issue arose when AggregateJoinTransposeRule was enabled, it is not an exclusive problem of the interaction between these two rules, but rather a problem on how AggregateProjectMergeRule was handling GroupingSets, right?

          Show
          jcamachorodriguez Jesus Camacho Rodriguez added a comment - LGTM, including the new methods that were added to ImmutableBitSet. Julian Hyde , for sake of clarity, although this issue arose when AggregateJoinTransposeRule was enabled, it is not an exclusive problem of the interaction between these two rules, but rather a problem on how AggregateProjectMergeRule was handling GroupingSets, right?
          Hide
          julianhyde Julian Hyde added a comment -

          Correct. There was no bug in AggregateJoinTransposeRule.

          I made some changes to the "Add a project if the group set is not in the same order or contains duplicates." section which were mostly just changing formatting. When I changed

          if (!newGroupSet.toList().equals(newKeys))

          to

          if (!newKeys.equals(aggregate.getGroupSet().asList()))

          it was not an improvement, except that it removed a warning about comparing lists of different types. I'll change that section back.

          A permuting project may be required

          Show
          julianhyde Julian Hyde added a comment - Correct. There was no bug in AggregateJoinTransposeRule. I made some changes to the "Add a project if the group set is not in the same order or contains duplicates." section which were mostly just changing formatting. When I changed if (!newGroupSet.toList().equals(newKeys)) to if (!newKeys.equals(aggregate.getGroupSet().asList())) it was not an improvement, except that it removed a warning about comparing lists of different types. I'll change that section back. A permuting project may be required
          Show
          julianhyde Julian Hyde added a comment - Fixed in http://git-wip-us.apache.org/repos/asf/incubator-calcite/commit/a42b88d4 .
          Hide
          julianhyde Julian Hyde added a comment -

          Resolved in release 1.3.0-incubating (2015-05-30).

          Show
          julianhyde Julian Hyde added a comment - Resolved in release 1.3.0-incubating (2015-05-30).

            People

            • Assignee:
              julianhyde Julian Hyde
              Reporter:
              julianhyde Julian Hyde
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development