[SPARK-49074] Fix cached Variant with column size greater than 128KB or individual variant larger than 2kb

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Richard Chen
• Reporter: Richard Chen
• Created: 31/Jul/24 19:40
• Updated: 01/Aug/24 00:48
• Resolved: 01/Aug/24 00:48

Description

Currently, the `actualSize` method of the `VARIANT` `columnType` isn't overridden, so we use the default size of 2kb for the `actualSize`. We should define `actualSize` so the cached variant column can correctly be written to the byte buffer

Issue Links

• links to: GitHub Pull Request #47559