[SPARK-48587] Avoid storage amplification when accessing sub-Variant

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: David Cashman
• Reporter: David Cashman
• Created: 11/Jun/24 17:48
• Updated: 17/Jun/24 05:27
• Resolved: 17/Jun/24 05:27

Description

When a variant_get expression returns a Variant, or a nested type containing Variant, we just return the sub-slice of the Variant value along with the full metadata, even though most of the metadata is probably unnecessary to represent the value. This may be very inefficient if the value is then written to disk (e.g. shuffle file or parquet). We should instead rebuild the value with minimal metadata.

Issue Links

• links to: GitHub Pull Request #46941