[SPARK-17172] pyspak hiveContext can not create UDF: Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.6.2
• Component/s: PySpark

Environment

spark version: 1.6.2
python version: 3.4.2 (v3.4.2:ab2c023a9432, Oct 5 2014, 20:42:22)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]

• Assignee: Unassigned
• Reporter: Andrew Davidson
• Created: 20/Aug/16 17:46
• Updated: 22/Aug/16 18:49
• Resolved: 22/Aug/16 08:10

Description

from pyspark.sql import HiveContext
sqlContext = HiveContext(sc)

1. Define udf
   from pyspark.sql.functions import udf
   def scoreToCategory(score):
   if score >= 80: return 'A'
   elif score >= 60: return 'B'
   elif score >= 35: return 'C'
   else: return 'D'

udfScoreToCategory=udf(scoreToCategory, StringType())

throws exception

Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.
: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

Issue Links

• duplicates: SPARK-15506 only one notebook can define a UDF; java.sql.SQLException: Another instance of Derby may have already booted the database