[FLINK-14055] Add advanced function DDL syntax "USING JAR"

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.16.0
• Component/s: Table SQL / API
• Labels: auto-unassigned sprint

Release Note

In 1.16, we introduced the `CREATE FUNCTION ... USING JAR` syntax to support the dynamic loading of the UDF jar in per job, which is convenient for platform developers to easily achieve UDF management. In addition, we also port the `ADD JAR` syntax from SqlClient to `TableEnvironment` side, this allows the syntax is more general to Table API users. However, due to inconsistent classloader in StreamExecutionEnvironment and TableEnvironment, the `ADD JAR` syntax is not available for Table API program currently, it will be resolved by https://issues.apache.org/jira/browse/FLINK-29240.



More information about this feature could be found in https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-function.

• Assignee: dalongliu
• Reporter: Bowen Li
• Created: 11/Sep/19 18:56
• Updated: 28/Nov/22 06:09
• Resolved: 19/Sep/22 14:18

Description

As FLIP-214  propose, this ticket is to support dynamically loading functions from external source in function DDL with advanced syntax like:

 

CREATE [TEMPORARY|TEMPORARY SYSTEM] FUNCTION [IF NOT EXISTS] [catalog_name.db_name.]function_name AS class_name [LANGUAGE JAVA|SCALA|PYTHON] [USING JAR‘resource_path’ [, JAR ‘resource_path’]*]; 

Issue Links

• is blocked by: FLINK-7151 Support function DDL

Sub-Tasks

• Allow passing a ClassLoader to EnvironmentSettings: Closed
• Support CREATE FUNCTION USING JAR syntax: Closed
• FlinkUserCodeClassLoader expose addURL method to allow to register jar dynamically: Closed
• Planner support to use jar which is registered by "USING JAR" syntax: Closed
• Table API support create function using customed jar: Closed
• Port ADD JAR /SHOW JARS syntax implementation from SqlClient to TableEnvironment side: Closed
• Introduce UserResourceManager to manage user defined resource: Closed
• Introduce MutableURLClassLoader as parent class of FlinkUserClassLoader and SafetyNetWrapperClassLoader: Closed
• StreamExecutionEnvironment configure method support override pipeline.jars option: Closed
• HiveFunctionDefinitionFactory load udf using user classloader instead of thread context classloader: Closed
• Use setting instead of merging for pipeline.jars in StreamExecutionEnvironment.configure: Closed
• Add document for CREATE FUNCTION USING JAR feature: Closed
• Updating Jar statement document: Closed
• ExpressionReducer does not work with jar resources: Closed