Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-1524

User defined functions in Scala cannot be applied

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.6.1
    • Component/s: None
    • Labels:

      Description

      Related to stackoverflow question http://stackoverflow.com/questions/39857803/class-not-found-when-applying-udf-in-apache-zeppelin
      and issue ZEPPELIN-150

      Applying an UDF in the scala interpreter generates a class not found exception.

      The following code works as expected in spark-shell (both 1.6.0 and 2.0.0) and can be used to reproduce the issue in Zeppelin 0.6.0:

      import org.apache.spark.sql.functions._
      val data = sqlContext.read.json(sc.parallelize(Seq("

      {'location' : 'london'}

      ", "

      {'location': 'tokyo'}

      ")))
      val cities = Seq("london", "paris")
      val urbf = udf

      { city: String => if (cities.contains(city)) 1 else 0 }

      sqlc.udf.register("sentiment", sentiment _)
      data.select($"location", urbf($"location")).show

      The error is:
      java.lang.ClassNotFoundException: $iwC$$iwC$$iwC$$iwC$$iwC$$$$cd8ca64e19a07f9732e2b3de5fd5fc88$$$$$iwC$$iwC$$iwC$$iwC$$anonfun$1

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Valur Valentin Ursu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: