Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-15592

Streaming sql throw hive exception when it doesn't use any hive table

    XMLWordPrintableJSON

Details

    Description

      I use the following streaming sql to query a kafka table whose metadata is store in hive metastore via HiveCatalog. But it will throw hive related exception which is very confusing.

      SQL

      SELECT *
FROM (
   SELECT *,
     ROW_NUMBER() OVER(
       ORDER BY event_ts) AS rownum
   FROM source_kafka)
WHERE rownum <= 10

      Exception

      Caused by: org.apache.flink.table.api.ValidationException: SQL validation failed. java.lang.reflect.InvocationTargetException
	at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:130)
	at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:105)
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:127)
	at org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:66)
	at org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:464)
	at org.apache.zeppelin.flink.sql.AbstractStreamSqlJob.run(AbstractStreamSqlJob.java:103)
	... 13 more
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at org.apache.flink.table.planner.functions.utils.HiveFunctionUtils.invokeGetResultType(HiveFunctionUtils.java:77)
	at org.apache.flink.table.planner.functions.utils.HiveAggSqlFunction.lambda$createReturnTypeInference$0(HiveAggSqlFunction.java:82)
	at org.apache.calcite.sql.SqlOperator.inferReturnType(SqlOperator.java:470)
	at org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:437)
	at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:303)
	at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:219)
	at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5600)
	at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5587)
	at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1691)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1676)
	at org.apache.calcite.sql.SqlCallBinding.getOperandType(SqlCallBinding.java:237)
	at org.apache.calcite.sql.type.OrdinalReturnTypeInference.inferReturnType(OrdinalReturnTypeInference.java:40)
	at org.apache.calcite.sql.type.SqlTypeTransformCascade.inferReturnType(SqlTypeTransformCascade.java:54)
	at org.apache.calcite.sql.SqlOperator.inferReturnType(SqlOperator.java:470)
	at org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:437)
	at org.apache.calcite.sql.SqlOverOperator.deriveType(SqlOverOperator.java:86)
	at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5600)
	at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5587)
	at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1691)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1676)
	at org.apache.calcite.sql.SqlAsOperator.deriveType(SqlAsOperator.java:133)
	at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5600)
	at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5587)
	at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1691)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1676)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:479)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4105)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3389)
	at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
	at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1008)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:968)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3122)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3104)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3376)
	at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
	at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1008)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:968)
	at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:216)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:943)
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:650)
	at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:126)
	... 18 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.flink.table.planner.functions.utils.HiveFunctionUtils.invokeGetResultType(HiveFunctionUtils.java:73)
	... 63 more
Caused by: org.apache.flink.table.functions.hive.FlinkHiveUDFException: Failed to get Hive result type from org.apache.hadoop.hive.ql.udf.generic.GenericUDAFRowNumber
	at org.apache.flink.table.functions.hive.HiveGenericUDAF.getHiveResultType(HiveGenericUDAF.java:202)
	... 68 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Only COMPLETE mode supported for row_number function
	at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFRowNumber$GenericUDAFAbstractRowNumberEvaluator.init(GenericUDAFRowNumber.java:100)
	at org.apache.flink.table.functions.hive.HiveGenericUDAF.init(HiveGenericUDAF.java:93)
	at org.apache.flink.table.functions.hive.HiveGenericUDAF.getHiveResultType(HiveGenericUDAF.java:196)
	... 68 more
ERROR   
Took 2 sec. Last updated by anonymous at January 15 2020, 11:12:16 AM.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. FLINK-15595 Entirely implement resolution order as FLIP-68 concept

          • Not a Priority -
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-15593 add doc to remind users not using Hive aggregate functions in streaming mode

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #10894

          Activity

            People

              lirui Rui Li
              zjffdu Jeff Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m