[IMPALA-7877] Support Hive GenericUDF

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: Impala 3.0
• Component/s: Backend
• Epic Color: ghx-label-1
• Assignee: Unassigned
• Reporter: eugen yushin
• Created: 21/Nov/18 14:07
• Updated: 02/Jun/22 01:41
• Resolved: 23/Dec/20 01:13

Description

Running Hive UDF extending GenericUDF interface results in class cast exception. Relevant code block:

LOG.debug("Loading UDF '" + udfPath + "' from " + jarPath);
loader = getClassLoader(jarPath);
Class<?> c = Class.forName(udfPath, true, loader);
Class<? extends UDF> udfClass = c.asSubclass(UDF.class);

Reproduce steps:

create function my_lower(string) returns string location '/path/to/hive-exec-1.1.0-cdh5.15.0.jar' symbol='org.apache.hadoop.hive.ql.udf.generic.GenericUDFLower';
select my_lower('Some String NOT ALREADY LOWERCASE');

Stack trace:

I1121 11:58:29.509138 29092 Frontend.java:952] Analyzing query: select my_lower('Some String NOT ALREADY LOWERCASE')
I1121 11:58:29.513121 29092 UdfExecutor.java:581] Loading UDF 'org.apache.hadoop.hive.ql.udf.generic.GenericUDFLower' from /var/lib/impala/udfs/hive-exec-1.1.0-cdh5.15.0.83728.2.jar
I1121 11:58:29.515535 29092 jni-util.cc:230] java.lang.ClassCastException: class org.apache.hadoop.hive.ql.udf.generic.GenericUDFLower
        at java.lang.Class.asSubclass(Class.java:3404)
        at org.apache.impala.hive.executor.UdfExecutor.init(UdfExecutor.java:584)
        at org.apache.impala.hive.executor.UdfExecutor.<init>(UdfExecutor.java:217)
        at org.apache.impala.service.FeSupport.NativeEvalExprsWithoutRow(Native Method)
        at org.apache.impala.service.FeSupport.EvalExprsWithoutRow(FeSupport.java:208)
        at org.apache.impala.service.FeSupport.EvalExprWithoutRow(FeSupport.java:163)
        at org.apache.impala.analysis.LiteralExpr.create(LiteralExpr.java:184)
        at org.apache.impala.rewrite.FoldConstantsRule.apply(FoldConstantsRule.java:68)
        at org.apache.impala.rewrite.ExprRewriter.applyRuleBottomUp(ExprRewriter.java:85)
        at org.apache.impala.rewrite.ExprRewriter.applyRuleRepeatedly(ExprRewriter.java:71)
        at org.apache.impala.rewrite.ExprRewriter.rewrite(ExprRewriter.java:55)
        at org.apache.impala.analysis.SelectList.rewriteExprs(SelectList.java:97)
        at org.apache.impala.analysis.SelectStmt.rewriteExprs(SelectStmt.java:894)
        at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:432)
        at org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:393)
        at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:962)
        at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:156)
I1121 11:58:29.523166 29092 status.cc:125] ClassCastException: class org.apache.hadoop.hive.ql.udf.generic.GenericUDFLower
    @           0x96663a  impala::Status::Status()
    @           0xcedfdd  impala::JniUtil::GetJniExceptionMsg()
    @          0x109457f  impala::HiveUdfCall::OpenEvaluator()
    @           0x96d757  impala::ScalarExprEvaluator::Open()
    @           0xbedc2d  Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow
    @     0x7fc705b49e6d  (unknown)

Marked as bug because there're no any notes related to this behaviour in docs (while it claims Impala supports Hive UDF, it should support all possible Hive UDF formats if other is not specified).

Issue Links

• duplicates: IMPALA-11162 Provide support for Hive Generic UDFs
• is related to: IMPALA-1393 Java UDFs: Support org.apache.hadoop.io package writables to be consistent with Hive
• is related to: IMPALA-9271 Provide UDF framework corresponding to Hive's GenericUDF