Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-6163

The spark ARRAY function gives exception when elements contain row(struct) type and NULL

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.36.0
    • 1.37.0
    • core
    • None

    Description

      when we run

      array(row(1,2), null)
      

      It will give 'Parameters must be of the same type' exception.

      org.apache.calcite.runtime.CalciteContextException: Parameters must be of the same type
          at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
          at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
          at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:507)
          at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:948)
          at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:933)
          at org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:5517)
          at org.apache.calcite.sql.SqlCallBinding.newValidationError(SqlCallBinding.java:414)
          at org.apache.calcite.sql.type.SameOperandTypeChecker.checkOperandTypesImpl(SameOperandTypeChecker.java:100)
          at org.apache.calcite.sql.type.SameOperandTypeChecker.checkOperandTypes(SameOperandTypeChecker.java:59)
          at org.apache.calcite.sql.SqlOperator.checkOperandTypes(SqlOperator.java:761)
          at org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:498)
          at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:347)
          at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:231)
          at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6575)
          at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6562)
          at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:166)
          at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1926)
          at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1911)
          at org.apache.calcite.sql.SqlNode.validateExpr(SqlNode.java:276)
          at org.apache.calcite.sql.SqlOperator.validateCall(SqlOperator.java:475)
          at org.apache.calcite.sql.validate.SqlValidatorImpl.validateCall(SqlValidatorImpl.java:6232)
          at org.apache.calcite.sql.SqlCall.validate(SqlCall.java:143)
          at org.apache.calcite.sql.SqlNode.validateExpr(SqlNode.java:275)
          at org.apache.calcite.sql.SqlOperator.validateCall(SqlOperator.java:475)
          at org.apache.calcite.sql.validate.SqlValidatorImpl.validateCall(SqlValidatorImpl.java:6232)
          at org.apache.calcite.sql.SqlCall.validate(SqlCall.java:143)
          at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1091)
          at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:797)
          at org.apache.calcite.sql.test.AbstractSqlTester.parseAndValidate(AbstractSqlTester.java:161)
          at org.apache.calcite.sql.test.AbstractSqlTester.validateAndThen(AbstractSqlTester.java:249)
          at org.apache.calcite.test.SqlOperatorFixtureImpl.lambda$forEachQueryValidateAndThen$1(SqlOperatorFixtureImpl.java:154)
          at org.apache.calcite.sql.test.AbstractSqlTester.forEachQuery(AbstractSqlTester.java:446)
          at org.apache.calcite.test.SqlOperatorFixtureImpl.forEachQueryValidateAndThen(SqlOperatorFixtureImpl.java:153)
          at org.apache.calcite.test.SqlOperatorFixtureImpl.checkType(SqlOperatorFixtureImpl.java:130)
          at org.apache.calcite.sql.test.SqlOperatorFixture.checkScalar(SqlOperatorFixture.java:226)
          at org.apache.calcite.test.SqlOperatorTest.testArrayFunction(SqlOperatorTest.java:10551)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.base/java.lang.reflect.Method.invoke(Method.java:566)
          at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
          at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
          at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
          at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
          at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
          at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
          at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
          at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
          at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
          at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
          at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
          at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
          at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
          at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
          at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
          at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
          at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
          at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
          at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
          at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
          at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
          at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
          at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
          at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
          at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
          at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
          at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
          at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
          at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
      

      however, the calcite std array constructor and spark array, it shows like below:

      0: jdbc:calcite:model=src/test/resources/mode> select array[row(1,2), null];
      +----------------+
      |     EXPR$0     |
      +----------------+
      | [{1, 2}, null] |
      +----------------+
      
      spark-sql (default)> select array((1,2), null);
      [{"col1":1,"col2":2},null]
      

      They supported row and null literal to coexist(the generated component type is nullable row/struct).

      The reason is because of current SameOperandTypeChecker of spark array will call SqlTypeUtil.isComparable like below:

      if (type1.isStruct() != type2.isStruct()) {return false;}
      

      It will reject row(struct) type and NULL, then throw exception. It seems we need a more general OperandTypeChecker here.

      Attachments

        Issue Links

          Activity

            taoran Ran Tao added a comment - Fixed via  https://github.com/apache/calcite/commit/e066266dcde21b3a0de90ec601ff539be1b8d7a3  

            Resolved in release 1.37.0 (2024-05-06)

            Sergey Nuyanzin Sergey Nuyanzin added a comment - Resolved in release 1.37.0 (2024-05-06)

            People

              taoran Ran Tao
              taoran Ran Tao
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: