[ZEPPELIN-4116] SQL UDF do not work with Spark New Interpreter

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 0.8.1
• Component/s: zeppelin-interpreter

Environment

Zeppelin 0.8.1.  Spark 2.2.2 . RHEL 7.

• Assignee: Unassigned
• Reporter: Mark Bidewell
• Created: 10/Apr/19 13:24
• Updated: 27/Apr/19 13:24
• Resolved: 26/Apr/19 14:43

Description

If I define a Spark UDF such as the following:

import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.functions._
import spark.implicits._

val data = sqlContext.range(0, 5)
val testudf = udf { (num: Integer) => num * num }
val udfcol = data.withColumn("id_square", testudf($"id"))

udfcol.show()

 

With the new interpreter, I randomly get an error similar to SPARK-9219.  Setting zeppelin.spark.useNew to false appears to consistently work correctly

Issue Links

• blocks: ZEPPELIN-4098 0.8.2 Release
• is a clone of: ZEPPELIN-4115 Case classes do not work with Spark New Interpreter