Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22965

Add deterministic parameter to registerJavaFunction

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: PySpark
    • Labels:
      None

      Description

      To register a JAVA UDF in PySpark, users are unable to specify the registered UDF is not deterministic. The proposal is to add the extra parameter deterministic at the end of the function registerJavaFunction

      Below is an example.

      >>> from pyspark.sql.types import DoubleType
      >>> sqlContext.registerJavaFunction("javaRand",
      ...   "test.org.apache.spark.sql.JavaRandUDF", DoubleType(), deterministic=False)
      >>> sqlContext.sql("SELECT javaRand(3)").collect()  # doctest: +SKIP
      [Row(UDF:javaRand(3)=3.12345)]
      

        Attachments

          Activity

            People

            • Assignee:
              smilegator Xiao Li
              Reporter:
              smilegator Xiao Li
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: