[SPARK-48067] Fix Variant default columns for more complex default variants

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Richard Chen
• Reporter: Richard Chen
• Created: 01/May/24 01:28
• Updated: 02/May/24 19:22
• Resolved: 02/May/24 19:22

Description

Default columns are stored as structfield metadata (string -> string) map. This means the literal values are stored as strings.

However, the string representation of a variant is the JSONified string. Thus, we need to wrap the string with `parse_json` to correctly use the default values

Issue Links

• links to: GitHub Pull Request #46312