[SPARK-4476] Use MapType for dict in json which has unique keys in each row.

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Done
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Davies Liu
• Created: 18/Nov/14 20:19
• Updated: 12/Dec/22 18:10
• Resolved: 04/Jan/19 14:26

Description

For the jsonRDD like this:

""" {a: 1} """
""" {b: 2} """
""" {c: 3} """
""" {d: 4} """
""" {e: 5} """

It will create a StructType with 5 fileds in it, each field come from a different row. It will be a problem if the RDD is large. A StructType with thousands or millions fields is hard to play with (will cause stack overflow during serialization).

It should be MapType for this case. We need a clear rule to decide StructType or MapType will be used for dict in json data.

cc yhuai marmbrus

Issue Links

• is duplicated by: SPARK-5936 Automatically convert a StructType to a MapType when the number of fields exceed a threshold.
• relates to: SPARK-4190 Allow users to provide transformation rules at JSON ingest
• links to: [Github] Pull Request #3406 (yhuai)

Sub-Tasks

• Accept MapType in the schema provided to a JSON dataset.: Resolved
• Automatically convert a StructType to a MapType when the number of fields exceed a threshold.: Resolved