[SPARK-41880] Function `from_json` should support non-literal expression

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Connect, PySpark
• Assignee: Ruifeng Zheng
• Reporter: Ruifeng Zheng
• Created: 04/Jan/23 07:16
• Updated: 08/Jan/23 01:16
• Resolved: 08/Jan/23 01:16

Description

File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1351, in pyspark.sql.connect.functions.from_json
Failed example:
    df.select(from_json(df.value, schema).alias("json")).collect()
Exception raised:
    Traceback (most recent call last):
      File "/Users/ruifeng.zheng/.dev/miniconda3/envs/spark_dev/lib/python3.9/doctest.py", line 1334, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.functions.from_json[12]>", line 1, in <module>
        df.select(from_json(df.value, schema).alias("json")).collect()
      File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/dataframe.py", line 1019, in collect
        pdf = self.toPandas()
      File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/dataframe.py", line 1054, in toPandas
        return self._session.client.to_pandas(query)
      File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client.py", line 413, in to_pandas
        return self._execute_and_fetch(req)
      File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client.py", line 573, in _execute_and_fetch
        self._handle_error(rpc_error)
      File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client.py", line 625, in _handle_error
        raise SparkConnectException(status.message) from None
    pyspark.sql.connect.client.SparkConnectException: Schema in from_json should be literal string, but got 'schema_of_json({"a": 0})

Issue Links

• links to: [Github] Pull Request #39444 (zhengruifeng)