Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41283 Feature parity: Functions API in Spark Connect
  3. SPARK-41880

Function `from_json` should support non-literal expression

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • Connect, PySpark
    • None

    Description

      File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1351, in pyspark.sql.connect.functions.from_json
      Failed example:
          df.select(from_json(df.value, schema).alias("json")).collect()
      Exception raised:
          Traceback (most recent call last):
            File "/Users/ruifeng.zheng/.dev/miniconda3/envs/spark_dev/lib/python3.9/doctest.py", line 1334, in __run
              exec(compile(example.source, filename, "single",
            File "<doctest pyspark.sql.connect.functions.from_json[12]>", line 1, in <module>
              df.select(from_json(df.value, schema).alias("json")).collect()
            File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/dataframe.py", line 1019, in collect
              pdf = self.toPandas()
            File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/dataframe.py", line 1054, in toPandas
              return self._session.client.to_pandas(query)
            File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client.py", line 413, in to_pandas
              return self._execute_and_fetch(req)
            File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client.py", line 573, in _execute_and_fetch
              self._handle_error(rpc_error)
            File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client.py", line 625, in _handle_error
              raise SparkConnectException(status.message) from None
          pyspark.sql.connect.client.SparkConnectException: Schema in from_json should be literal string, but got 'schema_of_json({"a": 0})
      
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            podongfeng Ruifeng Zheng
            podongfeng Ruifeng Zheng
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment