[SPARK-40820] Creating StructType from Json

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: PySpark, Spark Core
• Labels: pull-request-available

Language

Python Scala

• Assignee: Anthony Wainer Cachay Guivin
• Reporter: Anthony Wainer Cachay Guivin
• Created: 17/Oct/22 13:52
• Updated: 27/Oct/23 07:34
• Resolved: 27/Oct/23 07:34

Description

When create a StructType from a Python dictionary you use StructType.fromJson or in scala DataType.fromJson

To create a schema can be created as follows from the code below, but it requires to put inside the json: Nullable and Metadata, this is inconsistent because within the DataType class this by default.

schema = {
     "name": "name", "type": "string" 
}

StructField.fromJson(schema)

Python Error:

from pyspark.sql.types import StructField

schema = {
     "name": "c1", "type": "string" 
}

StructField.fromJson(schema)

>>
Traceback (most recent call last):
File "code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
File "pyspark/sql/types.py", line 583, in fromJson
json["nullable"],
KeyError: 'nullable' 

Scala Error:

val schema = """
        |{
        |    "type": "struct",
        |    "fields": [
        |        {
        |            "name": "c1",
        |            "type": "string",
        |            "nullable": false
        |        }
        |    ]
        |}
        |""".stripMargin

DataType.fromJson(schema)

>>
Failed to convert the JSON string '{"name":"c1","type":"string"}' to a field.
java.lang.IllegalArgumentException: Failed to convert the JSON string '{"name":"c1","type":"string"}' to a field.
at org.apache.spark.sql.types.DataType$.parseStructField(DataType.scala:268)
at org.apache.spark.sql.types.DataType$.$anonfun$parseDataType$1(DataType.scala:225)

Issue Links

• links to: [Github] Pull Request #38285 (anthonywainer)
• links to: [Github] Pull Request #38285 (anthonywainer)
• links to: GitHub Pull Request #43474