Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.0.0, 2.0.1, 2.0.2
-
None
-
HDP: 2.5
Spark: 2.0.0
Description
Spark Thriftserver is unable to run a HiveUDTF.
It throws the error that it is unable to find the functions although the function registration succeeds and the funtions does show up in the list output by show functions.
I am using a Hive UDTF, registering it using a jar placed on my local machine. Calling it using the following commands:
//Registering the functions, this command succeeds.
CREATE FUNCTION SampleUDTF AS 'com.fuzzylogix.experiments.udf.hiveUDF.SampleUDTF' USING JAR '/root/spark_files/experiments-1.2.jar';
//Thriftserver is able to look up the functuion, on this command:
DESCRIBE FUNCTION SampleUDTF;
Output: +-----------------------------------------------------------+--+ | function_desc | +-----------------------------------------------------------+--+ | Function: default.SampleUDTF | | Class: com.fuzzylogix.experiments.udf.hiveUDF.SampleUDTF | | Usage: N/A. | +-----------------------------------------------------------+--+
// Calling the function:
SELECT SampleUDTF('Paris');
Output of the above command: Error: org.apache.spark.sql.AnalysisException: Undefined function: 'SampleUDTF'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 7 (state=,code=0)
I have also tried with using a non-local (on hdfs) jar, but I get the same error.
My environment: HDP 2.5 with spark 2.0.0
I have attached the class file for the UDTF I am using in testing this.