Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-4054

RepeatUnion containing a Correlate with a transientScan on its RHS causes NPE

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.23.0
    • Fix Version/s: None
    • Component/s: core
    • Labels:
      None

      Description

      This is not a very urgent problem, since this is an exception on an operator tagged as "Experimental", and several "not-so-usual" circumstances must occur in order to reach the NPE:

      • We need a "recursive plan", i.e. a RepeatUnion with a TransientScan (see examples in EnumerableRepeatUnionTest.java).
      • Inside the repeat union, there must be a Correlate, or a Join that gets implemented as a Correlate due to JoinToCorrelateRule.
      • We have the TransientScan (i.e. the scan on the transient table) on the right-hand-side of the correlate/join. Normally, transientScan always appear on the LHS of a correlate/join (because when the plan is built they need the top of the RelBuilder's stack to get its rowType); however, it can get on the RHS due to JoinCommuteRule.

      We can force these conditions and reproduce the issue with the following unit test (in EnumerableRepeatUnionTest.java):

        @Test void testRepeatUnionWithCorrelateWithTransientScanOnItsRight() {
          CalciteAssert.that()
              .with(CalciteConnectionProperty.LEX, Lex.JAVA)
              .with(CalciteConnectionProperty.FORCE_DECORRELATE, false)
              .withSchema("s", new ReflectiveSchema(new HierarchySchema()))
              .query("?")
              .withHook(Hook.PLANNER, (Consumer<RelOptPlanner>) planner -> {
                planner.addRule(JoinToCorrelateRule.INSTANCE);
                planner.removeRule(JoinCommuteRule.INSTANCE);
                planner.removeRule(EnumerableRules.ENUMERABLE_MERGE_JOIN_RULE);
                planner.removeRule(EnumerableRules.ENUMERABLE_JOIN_RULE);
              })
              .withRel(builder -> {
                builder
                    //   WITH RECURSIVE delta(empid, name) as (
                    //     SELECT empid, name FROM emps WHERE empid = 2
                    //     UNION ALL
                    //     SELECT e.empid, e.name FROM delta d
                    //                            JOIN hierarchies h ON d.empid = h.managerid
                    //                            JOIN emps e        ON h.subordinateid = e.empid
                    //   )
                    //   SELECT empid, name FROM delta
                    .scan("s", "emps")
                    .filter(
                        builder.equals(
                            builder.field("empid"),
                            builder.literal(2)))
                    .project(
                        builder.field("emps", "empid"),
                        builder.field("emps", "name"))
      
                    .transientScan("#DELTA#");
                RelNode transientScan = builder.build(); // pop the transientScan to use it later
      
                builder
                    .scan("s", "hierarchies")
                    .push(transientScan) // use the transientScan as right input of the join
                    .join(
                        JoinRelType.INNER,
                        builder.equals(
                            builder.field(2, "#DELTA#", "empid"),
                            builder.field(2, "hierarchies", "managerid")))
      
                    .scan("s", "emps")
                    .join(
                        JoinRelType.INNER,
                        builder.equals(
                            builder.field(2, "hierarchies", "subordinateid"),
                            builder.field(2, "emps", "empid")))
                    .project(
                        builder.field("emps", "empid"),
                        builder.field("emps", "name"))
                    .repeatUnion("#DELTA#", true);
                return builder.build();
              })
              .explainHookMatches(""
                  + "EnumerableRepeatUnion(all=[true])\n"
                  + "  EnumerableTableSpool(readType=[LAZY], writeType=[LAZY], table=[[#DELTA#]])\n"
                  + "    EnumerableCalc(expr#0..4=[{inputs}], expr#5=[2], expr#6=[=($t0, $t5)], empid=[$t0], name=[$t2], $condition=[$t6])\n"
                  + "      EnumerableTableScan(table=[[s, emps]])\n"
                  + "  EnumerableTableSpool(readType=[LAZY], writeType=[LAZY], table=[[#DELTA#]])\n"
                  + "    EnumerableCalc(expr#0..8=[{inputs}], empid=[$t4], name=[$t6])\n"
                  + "      EnumerableCorrelate(correlation=[$cor1], joinType=[inner], requiredColumns=[{1}])\n"
                  // It is important to have EnumerableCorrelate + #DELTA# table scan on its right
                  + "        EnumerableCorrelate(correlation=[$cor0], joinType=[inner], requiredColumns=[{0}])\n"
                  + "          EnumerableTableScan(table=[[s, hierarchies]])\n"
                  + "          EnumerableCalc(expr#0..1=[{inputs}], expr#2=[$cor0], expr#3=[$t2.managerid], expr#4=[=($t0, $t3)], proj#0..1=[{exprs}], $condition=[$t4])\n"
                  + "            EnumerableInterpreter\n"
                  + "              BindableTableScan(table=[[#DELTA#]])\n"
                  + "        EnumerableCalc(expr#0..4=[{inputs}], expr#5=[$cor1], expr#6=[$t5.subordinateid], expr#7=[=($t6, $t0)], proj#0..4=[{exprs}], $condition=[$t7])\n"
                  + "          EnumerableTableScan(table=[[s, emps]])\n")
              .returnsUnordered(""
                  + "empid=2; name=Emp2\n"
                  + "empid=3; name=Emp3\n"
                  + "empid=5; name=Emp5");
        }
      

      Which fails with NPE on the dynamic generated code:

      Error while executing SQL "?": null
      java.sql.SQLException: Error while executing SQL "?": null
      	at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
      	at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
      	at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163)
      	at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
      	at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:532)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1511)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1450)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1509)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1499)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.returnsUnordered(CalciteAssert.java:1521)
      	at org.apache.calcite.test.enumerable.EnumerableRepeatUnionTest.testRepeatUnionWithCorrelateWithTransientScanOnItsRight(EnumerableRepeatUnionTest.java:296)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
      	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
      	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
      	at org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:46)
      	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:139)
      	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:131)
      	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:81)
      	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
      	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
      	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
      	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
      	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
      	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
      	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
      	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
      	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
      	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
      	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
      	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
      	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
      	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
      	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
      	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
      	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
      	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
      	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
      	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
      	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
      	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:171)
      	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:115)
      	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
      	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
      	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
      	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
      	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
      	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
      	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
      	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
      	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:171)
      	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:115)
      	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
      	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
      	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
      	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
      	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
      	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
      	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
      	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
      	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:171)
      	at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
      	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
      	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
      	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
      	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
      Caused by: java.lang.NullPointerException
      	at Baz.bind(Unknown Source)
      	at org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:355)
      	at org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:315)
      	at org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:507)
      	at org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:498)
      	at org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:182)
      	at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:64)
      	at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:43)
      	at org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
      	at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:567)
      	at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
      	at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
      	... 66 more
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                rubenql Ruben Q L
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: