[SPARK-18832] Spark SQL: Thriftserver unable to run a registered Hive UDTF

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.0.0, 2.0.1, 2.0.2
• Component/s: SQL
• Labels: bulk-closed

Environment

HDP: 2.5
Spark: 2.0.0

• Assignee: Unassigned
• Reporter: Lokesh Yadav
• Created: 12/Dec/16 19:31
• Updated: 25/May/21 01:52
• Resolved: 25/May/21 01:42

Description

Spark Thriftserver is unable to run a HiveUDTF.
It throws the error that it is unable to find the functions although the function registration succeeds and the funtions does show up in the list output by show functions.

I am using a Hive UDTF, registering it using a jar placed on my local machine. Calling it using the following commands:

//Registering the functions, this command succeeds.
CREATE FUNCTION SampleUDTF AS 'com.fuzzylogix.experiments.udf.hiveUDF.SampleUDTF' USING JAR '/root/spark_files/experiments-1.2.jar';

//Thriftserver is able to look up the functuion, on this command:
DESCRIBE FUNCTION SampleUDTF;

Output: 
+-----------------------------------------------------------+--+
|                       function_desc                       |
+-----------------------------------------------------------+--+
| Function: default.SampleUDTF                              |
| Class: com.fuzzylogix.experiments.udf.hiveUDF.SampleUDTF  |
| Usage: N/A.                                               |
+-----------------------------------------------------------+--+

// Calling the function:
SELECT SampleUDTF('Paris');

Output of the above command: Error: org.apache.spark.sql.AnalysisException: Undefined function: 'SampleUDTF'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 7 (state=,code=0)

I have also tried with using a non-local (on hdfs) jar, but I get the same error.

My environment: HDP 2.5 with spark 2.0.0

I have attached the class file for the UDTF I am using in testing this.