Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7877

Support Hive GenericUDF

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: Impala 3.0
    • Fix Version/s: None
    • Component/s: Backend
    • Labels:
      None
    • Epic Color:
      ghx-label-1

      Description

      Running Hive UDF extending GenericUDF interface results in class cast exception. Relevant code block:

      LOG.debug("Loading UDF '" + udfPath + "' from " + jarPath);
      loader = getClassLoader(jarPath);
      Class<?> c = Class.forName(udfPath, true, loader);
      Class<? extends UDF> udfClass = c.asSubclass(UDF.class);
      

      Reproduce steps:

      create function my_lower(string) returns string location '/path/to/hive-exec-1.1.0-cdh5.15.0.jar' symbol='org.apache.hadoop.hive.ql.udf.generic.GenericUDFLower';
      select my_lower('Some String NOT ALREADY LOWERCASE');
      

      Stack trace:

      I1121 11:58:29.509138 29092 Frontend.java:952] Analyzing query: select my_lower('Some String NOT ALREADY LOWERCASE')
      I1121 11:58:29.513121 29092 UdfExecutor.java:581] Loading UDF 'org.apache.hadoop.hive.ql.udf.generic.GenericUDFLower' from /var/lib/impala/udfs/hive-exec-1.1.0-cdh5.15.0.83728.2.jar
      I1121 11:58:29.515535 29092 jni-util.cc:230] java.lang.ClassCastException: class org.apache.hadoop.hive.ql.udf.generic.GenericUDFLower
              at java.lang.Class.asSubclass(Class.java:3404)
              at org.apache.impala.hive.executor.UdfExecutor.init(UdfExecutor.java:584)
              at org.apache.impala.hive.executor.UdfExecutor.<init>(UdfExecutor.java:217)
              at org.apache.impala.service.FeSupport.NativeEvalExprsWithoutRow(Native Method)
              at org.apache.impala.service.FeSupport.EvalExprsWithoutRow(FeSupport.java:208)
              at org.apache.impala.service.FeSupport.EvalExprWithoutRow(FeSupport.java:163)
              at org.apache.impala.analysis.LiteralExpr.create(LiteralExpr.java:184)
              at org.apache.impala.rewrite.FoldConstantsRule.apply(FoldConstantsRule.java:68)
              at org.apache.impala.rewrite.ExprRewriter.applyRuleBottomUp(ExprRewriter.java:85)
              at org.apache.impala.rewrite.ExprRewriter.applyRuleRepeatedly(ExprRewriter.java:71)
              at org.apache.impala.rewrite.ExprRewriter.rewrite(ExprRewriter.java:55)
              at org.apache.impala.analysis.SelectList.rewriteExprs(SelectList.java:97)
              at org.apache.impala.analysis.SelectStmt.rewriteExprs(SelectStmt.java:894)
              at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:432)
              at org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:393)
              at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:962)
              at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:156)
      I1121 11:58:29.523166 29092 status.cc:125] ClassCastException: class org.apache.hadoop.hive.ql.udf.generic.GenericUDFLower
          @           0x96663a  impala::Status::Status()
          @           0xcedfdd  impala::JniUtil::GetJniExceptionMsg()
          @          0x109457f  impala::HiveUdfCall::OpenEvaluator()
          @           0x96d757  impala::ScalarExprEvaluator::Open()
          @           0xbedc2d  Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow
          @     0x7fc705b49e6d  (unknown)
      

      Marked as bug because there're no any notes related to this behaviour in docs (while it claims Impala supports Hive UDF, it should support all possible Hive UDF formats if other is not specified).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                eyushin eugen yushin
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: