[FLINK-20606] sql cli with hive catalog cannot create function using user classes from jar which specified by -j option

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.10.2, 1.11.2, 1.12.0
• Fix Version/s: 1.13.0
• Component/s: Connectors / Hive, Table SQL / API, Table SQL / Client
• Labels: pull-request-available
• Assignee: akisaya
• Reporter: akisaya
• Created: 14/Dec/20 23:24
• Updated: 28/May/21 06:59
• Resolved: 30/Dec/20 09:26

Description

with flink version 1.12.0(versions before also affected)

I started a sql cli  with a hive catalog and specified a user jar file with -j option like this:

bin/sql-client.sh embedded -j /Users/akis/Desktop/flink-func/myfunc.jar

when i tried to create a custom function using class from myfunc.jar，cli reported ClassNotFoundException.

 

Flink SQL> use catalog myhive;

Flink SQL> create function myfunc1 as 'me.aki.flink.flinkudf.MyFunc';
[ERROR] Could not execute SQL statement. Reason:
java.lang.ClassNotFoundException: me.aki.flink.flinkudf.MyFunc

 

 

me.aki.flink.flinkudf.MyFunc is the identifier of udf，which defined like this

 

package me.aki.flink.flinkudf;

import org.apache.flink.table.functions.ScalarFunction;

public class MyFunc extends ScalarFunction {
    public String eval(String s) {
        return "myfunc_" + s;
    }
}

 

 

 

after walking through the related code, I believe this is a bug caused by wrong classloader

 

when using a hive catalog, flink will use  CatalogFunctionImpl  to wrap the function。 The

isGeneric() methed  uses Class.forName(String clazzName) which will use a current classloader(classloader loads flink/lib) to determine the class。

 

however with -j option, user jar is set to the ExecutionContext and loaded by another userClassLoader

 

and the fix can be easy to pass a classloader to the Class.forName method.

ClassLoader cl = Thread.currentThread().getContextClassLoader();
Class c = Class.forName(className, true, cl);

after do such fix and build a new flink dist，create function behaves right

 

Flink SQL> select myfunc1('1');

// output
     EXPR$0
     myfunc_1

 

 

 

 

 

 

 

Issue Links

• fixes: FLINK-22632 CatalogFunctionImpl.isGeneric should use ContextClassLoader
• links to: GitHub Pull Request #14392