[SPARK-45891] Support Variant data type

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Chenhao Li
• Reporter: Chenhao Li
• Created: 10/Nov/23 19:23
• Updated: 19/Aug/24 03:12
• Resolved: 18/Mar/24 03:08

Description

I propose to add a Variant data type in Spark. It is used to efficiently represent semi-structured values without a user-specified schema. Currently, many users are depending on JSON expressions to handle JSON data, which can often lead to repeated JSON parsing and degraded performance. One of the major goals of the Variant type is to use a more efficient binary representation internally and avoid repeated JSON parsing. At the same time, it keeps the flexibility of schemaless JSON data.

Issue Links

• links to: GitHub Pull Request #45539
• links to: GitHub Pull Request #46311
• links to: GitHub Pull Request #46831
• links to: GitHub Pull Request #47473
• links to: GitHub Pull Request #47792

Sub-Tasks

• Add Golden Table Tests for Variant from different engines: Open
• Functions to shred a Variant into components: Open
• Add support for interval types in the Variant spec: Open
• Add variant metrics to JSON Scan nodes and Project nodes containing variant-constructor expressions: Open