[CALCITE-4317] Some rules fail to handle Aggregate node if RelFieldTrimmer trims all the fields

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.27.0
• Component/s: core
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Rafay A
• Created: 07/Oct/20 19:04
• Updated: 03/Jun/21 22:32
• Resolved: 29/Nov/20 10:20

Description

Consider this query:

select o_orderkey from 
(SELECT count(*) cnt_star, count(l_orderkey) cnt_ok FROM lineitem)
cross join orders
limit 10

 which generates this plan:

LogicalSort(fetch=[10])
  LogicalProject(o_orderkey=[$2])
    LogicalJoin(condition=[true], joinType=[inner])
      LogicalAggregate(group=[{}], cnt_star=[COUNT()], cnt_ok=[COUNT($0)])
        LogicalProject(l_orderkey=[$0])
          ScanCrel(table=[lineitem], columns=[`l_orderkey`, `l_partkey`, `l_suppkey`, `l_linenumber`, `l_quantity`, `l_extendedprice`, `l_discount`, `l_tax`, `l_returnflag`, `l_linestatus`, `l_shipdate`, `l_commitdate`, `l_receiptdate`, `l_shipinstruct`, `l_shipmode`, `l_comment`])
      ScanCrel(table=[orders], columns=[`o_orderkey`, `o_custkey`, `o_orderstatus`, `o_totalprice`, `o_orderdate`, `o_orderpriority`, `o_clerk`, `o_shippriority`, `o_comment`])

and after we apply RelFieldTrimmer:

LogicalSort(fetch=[10])
  LogicalProject(o_orderkey=[$0])
    LogicalJoin(condition=[true], joinType=[inner])
      LogicalAggregate(group=[{}])
        LogicalProject(l_orderkey=[$0])
          ScanCrel(table=[lineitem], columns=[`l_orderkey`])
      ScanCrel(table=[orders], columns=[`o_orderkey`])

 

We see that the LogicalAggregate has no groups/functions after applying the trimmer. When we apply ProjectJoinTransposeRule or LoptOptimizeJoinRule after trimming, we are seeing these exceptions:

For LoptOptimizeJoinRule:

  (java.lang.ArrayIndexOutOfBoundsException) 0
    com.google.common.collect.RegularImmutableList.get():75
    org.apache.calcite.rel.metadata.RelMdColumnOrigins.getColumnOrigins():77
    sun.reflect.NativeMethodAccessorImpl.invoke0():-2
    sun.reflect.NativeMethodAccessorImpl.invoke():62
    sun.reflect.DelegatingMethodAccessorImpl.invoke():43
    java.lang.reflect.Method.invoke():498
    org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke():178
    com.sun.proxy.$Proxy175.getColumnOrigins():-1
    sun.reflect.GeneratedMethodAccessor144.invoke():-1
    sun.reflect.DelegatingMethodAccessorImpl.invoke():43
    java.lang.reflect.Method.invoke():498
    org.apache.calcite.rel.metadata.ChainedRelMetadataProvider$ChainedInvocationHandler.invoke():139
    com.sun.proxy.$Proxy175.getColumnOrigins():-1
    sun.reflect.GeneratedMethodAccessor144.invoke():-1
    sun.reflect.DelegatingMethodAccessorImpl.invoke():43
    java.lang.reflect.Method.invoke():498
    com.sun.proxy.$Proxy175.getColumnOrigins():-1
    sun.reflect.GeneratedMethodAccessor144.invoke():-1
    sun.reflect.DelegatingMethodAccessorImpl.invoke():43
    java.lang.reflect.Method.invoke():498
    org.apache.calcite.rel.metadata.ChainedRelMetadataProvider$ChainedInvocationHandler.invoke():139
    com.sun.proxy.$Proxy175.getColumnOrigins():-1
    sun.reflect.GeneratedMethodAccessor144.invoke():-1
    sun.reflect.DelegatingMethodAccessorImpl.invoke():43
    java.lang.reflect.Method.invoke():498
    com.sun.proxy.$Proxy175.getColumnOrigins():-1
    sun.reflect.GeneratedMethodAccessor144.invoke():-1
    sun.reflect.DelegatingMethodAccessorImpl.invoke():43
    java.lang.reflect.Method.invoke():498
    org.apache.calcite.rel.metadata.RelMetadataQuery$MetadataInvocationHandler.invoke():110
    com.sun.proxy.$Proxy175.getColumnOrigins():-1
    org.apache.calcite.rel.metadata.RelMetadataQuery.getColumnOrigins():293
    org.apache.calcite.rel.metadata.RelMetadataQuery.getTableOrigin():348
    org.apache.calcite.rel.rules.LoptOptimizeJoinRule.getSimpleFactors():377
    org.apache.calcite.rel.rules.LoptOptimizeJoinRule.findRemovableSelfJoins():297
    org.apache.calcite.rel.rules.LoptOptimizeJoinRule.onMatch():126
    org.apache.calcite.plan.AbstractRelOptPlanner.fireRule():317
    org.apache.calcite.plan.hep.HepPlanner.applyRule():556
    org.apache.calcite.plan.hep.HepPlanner.applyRules():415
    org.apache.calcite.plan.hep.HepPlanner.executeInstruction():280
    org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute():74
    org.apache.calcite.plan.hep.HepPlanner.executeProgram():211
    org.apache.calcite.plan.hep.HepPlanner.findBestExp():198

For ProjectJoinTransposeRule:

  (java.lang.ArrayIndexOutOfBoundsException) 0
    com.google.common.collect.RegularImmutableList.get():75
    org.apache.calcite.rel.rules.PushProjector.createProjectRefsAndExprs():477
    org.apache.calcite.rel.rules.ProjectJoinTransposeRule.onMatch():138
    org.apache.calcite.plan.AbstractRelOptPlanner.fireRule():317
    org.apache.calcite.plan.hep.HepPlanner.applyRule():556
    org.apache.calcite.plan.hep.HepPlanner.applyRules():415
    org.apache.calcite.plan.hep.HepPlanner.executeInstruction():280
    org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute():74
    org.apache.calcite.plan.hep.HepPlanner.executeProgram():211
    org.apache.calcite.plan.hep.HepPlanner.findBestExp():198

I think we should either fix the trimmer, or the rules to handle these cases.

Issue Links

• blocks: CALCITE-4378 Include ProjectJoinTransposeRule in the main optimizer process
• is duplicated by: CALCITE-4381 CoreQuidemTest#[8] throws ArrayIndexOutOfBoundsException when enable ProjectJoinTransposeRule
• is related to: CALCITE-2720 Presence of an empty aggregation should not lead to Exceptions
• is related to: CALCITE-4334 LITERAL_AGG, an aggregate function that returns a constant value
• links to: GitHub Pull Request #2211