Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-1347 Release 0.6.2
  3. ZEPPELIN-1442

UDF can not be found due to 2 instances of SparkSession is created

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.1
    • Fix Version/s: 0.6.2, 0.7.0
    • Component/s: pySpark
    • Labels:
      None

      Description

      It worked in 0.5.6 with spark 1.6.2
      On 2.0 - I've checked - when I run just pySpark everything working fine.

      I've build zeppelin with

      mvn clean package -Pspark-2.0 -Phadoop-2.6 -Pyarn -Ppyspark -DskipTests
      
      %pyspark
      from pyspark.sql import SQLContext, Row
      from pyspark.sql.types import StringType, IntegerType, StructType, StructField, MapType, FloatType, ArrayType
      from pyspark.sql.functions import udf
      
      sqlContext.registerFunction("stringLengthString", lambda x: len(x))
      

      Returns:

      Traceback (most recent call last):
        File "/tmp/zeppelin_pyspark-3201872850141735060.py", line 266, in <module>
          raise Exception(traceback.format_exc())
      Exception: Traceback (most recent call last):
        File "/tmp/zeppelin_pyspark-3201872850141735060.py", line 264, in <module>
          exec(code)
        File "<stdin>", line 7, in <module>
        File "/home/egor/spark/python/pyspark/sql/context.py", line 203, in registerFunction
          self.sparkSession.catalog.registerFunction(name, f, returnType)
      AttributeError: 'JavaMember' object has no attribute 'registerFunction'
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zjffdu Jeff Zhang
                Reporter:
                epahomov Egor Pahomov
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: