Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-6304 Improve the Arrow adapter
  3. CALCITE-6429

Arrow adapter should default to the Enumerable convention for unsupported filters

    XMLWordPrintableJSON

Details

    Description

      Currently, the adapter fails with different errors when one of the known unsupported features is used, while it might simply catch the "UnsupportedOperationException" and bail out, thus allowing to generate a plan where the unsupported operation is implemented in the Enumerable convention.

       

      For instance, currently several tests are disabled for this reason, like 
      testArrowProjectFieldsWithDisjunctiveFilter, because it would fail as follows:

      Error while executing SQL "select "intField", "stringField"
      from arrowdata
      where "intField"=12 or "stringField"='12'": Error while applying rule ArrowFilterRule, args [rel#31:LogicalFilter.NONE.[](input=RelSubset#15,condition=OR(=($0, 12), =($1, '12'))), rel#1:ArrowTableScan.ARROW.[](table=[ARROW, ARROWDATA],fields=[0, 1, 2, 3])]
      java.sql.SQLException: Error while executing SQL "select "intField", "stringField"
      from arrowdata
      where "intField"=12 or "stringField"='12'": Error while applying rule ArrowFilterRule, args [rel#31:LogicalFilter.NONE.[](input=RelSubset#15,condition=OR(=($0, 12), =($1, '12'))), rel#1:ArrowTableScan.ARROW.[](table=[ARROW, ARROWDATA],fields=[0, 1, 2, 3])]
          at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
          at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
          at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:164)
          at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:228)
          at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:566)
          at org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1495)
          at org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1434)
          at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1493)
          at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1483)
          at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1446)
          at org.apache.calcite.adapter.arrow.ArrowAdapterTest.testArrowProjectFieldsWithDisjunctiveFilter(ArrowAdapterTest.java:260)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
          at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
          at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
          at org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
          at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
          at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
          at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
          at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
          at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
          at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
          at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
          at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
          at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
          at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
          at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
          at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
          at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
          at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
          at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
          at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
          at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
          at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
          at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
          at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
          at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
          at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
          at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
          at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
          at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
          at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
          at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
          at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
          at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
          at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
          at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
          at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
          at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
          at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
          at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
          Suppressed: org.apache.calcite.util.TestUtil$ExtraInformation: With materializationsEnabled=false, limit=-1
              at org.apache.calcite.util.TestUtil.rethrow(TestUtil.java:389)
              at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:598)
              ... 64 more
      Caused by: java.lang.RuntimeException: Error while applying rule ArrowFilterRule, args [rel#31:LogicalFilter.NONE.[](input=RelSubset#15,condition=OR(=($0, 12), =($1, '12'))), rel#1:ArrowTableScan.ARROW.[](table=[ARROW, ARROWDATA],fields=[0, 1, 2, 3])]
          at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:250)
          at org.apache.calcite.plan.volcano.IterativeRuleDriver.drive(IterativeRuleDriver.java:59)
          at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:524)
          at org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:282)
          at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:348)
          at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:177)
          at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314)
          at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:220)
          at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:666)
          at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:519)
          at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:487)
          at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:237)
          at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:702)
          at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:677)
          at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:157)
          ... 66 more
      Caused by: java.lang.UnsupportedOperationException: Unsupported disjunctive condition OR(=($0, 12), =($1, '12'))
          at org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
          at org.apache.calcite.adapter.arrow.ArrowFilter.<init>(ArrowFilter.java:43)
          at org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:103)
          at org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:88)
          at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223)
          ... 80 more

      The ideal plan the test should generate is:

      "PLAN=ArrowToEnumerableConverter\n"
      + " ArrowProject(intField=[$0], stringField=[$1])\n"
      + " ArrowFilter(condition=[OR(=($0, 12), =($1, '12'))])\n"
      + " ArrowTableScan(table=[[ARROW, ARROWDATA]], fields=[[0, 1, 2, 3]])\n\n";

      The test query, however, after the proposed workaround, could produce this valid plan in the meantime:

      PLAN=EnumerableCalc(expr#0..1=[{inputs}], expr#2=[12], "
      + "expr#3=[=($t0, $t2)], expr#4=['12':VARCHAR], expr#5=[=($t1, $t4)], "
      + "expr#6=[OR($t3, $t5)], proj#0..1=[{exprs}], $condition=[$t6])\n"
      + " ArrowToEnumerableConverter\n"
      + " ArrowProject(intField=[$0], stringField=[$1])\n"
      + " ArrowTableScan(table=[[ARROW, ARROWDATA]], fields=[[0, 1, 2, 3]])

      The advantage would be that more queries would be supported right away, while waiting for the missing features to be contributed.

      Attachments

        Issue Links

          Activity

            People

              asolimando Alessandro Solimando
              asolimando Alessandro Solimando
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: