Major Description
Inferring schema for struct columns causing schema issues as below.
data = [{"inside_struct": {"payment": 100.5, "name": "Lee"}}]
df = spark.createDataFrame(data)
df.show()
+--------------------+
| inside_struct|
+--------------------+
|{name -> null, pa...|
+--------------------+
The "inside_struct" is a map, and the "name" column inside of it becomes null.
The schema inferring might decide on a map type with a value type of the first field of the struct, we should fix it.
Attachments
Issue Links
- is duplicated by
-
SPARK-38839 Creating a struct with a float inside
-
- Resolved
-
- links to
