[SPARK-48175] Store collation information in metadata and not in type for SER/DE

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark, SQL
• Labels: pull-request-available
• Epic Link: Collation support
• Assignee: Stefan Kandic
• Reporter: Stefan Kandic
• Created: 07/May/24 15:25
• Updated: 09/Jul/24 20:11
• Resolved: 18/May/24 07:18

Description

Changing serialization and deserialization of collated strings so that the collation information is put in the metadata of the enclosing struct field - and then read back from there during parsing.

Issue Links

• links to: GitHub Pull Request #46280