[SPARK-46830] Introducing collation concept into Spark

• Type: Epic
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Spark Core
• Epic Name: Collation support
• Assignee: Unassigned
• Reporter: Aleksandar Tomic
• Created: 24/Jan/24 17:10
• Updated: 5 days ago 12:53

Description

This feature will introduce collation support to the Spark engine. This means that:

 

1. Every StringType will have an associated collation. Default remains UTF8 Binary, which will behave under the same rules as current UTF8 String comparison.
2. Collation will be respected in all collation sensitive operations - comparisons, hashing, string operations (contains, startWith, endsWith etc.)
3. Collation can be set through following ways:
   1. COLLATE expression. e.g. strExpr COLLATE collation_name
   2. In CREATE TABLE column definition
   3. By setting session collation.
4. All the Spark operators need to respect collation settings (filters, joins, shuffles, aggs etc.)

 

This is a high level description of the feature. You can find detailed design under this link (doc is in attachment as well).