[SPARK-47483] Add support for aggregation and join operations on arrays of collated strings

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Epic Link: Collation support
• Assignee: Nikola Mandic
• Reporter: Nikola Mandic
• Created: 20/Mar/24 08:33
• Updated: 22/Mar/24 07:15
• Resolved: 22/Mar/24 07:15

Description

Example of aggregation sequence:

create table t(a array<string collate utf8_binary_lcase>) using parquet;


insert into t(a) values(array('a' collate utf8_binary_lcase));
insert into t(a) values(array('A' collate utf8_binary_lcase));


select distinct a from t; 

Example of join sequence:

create table l(a array<string collate utf8_binary_lcase>) using parquet;
create table r(a array<string collate utf8_binary_lcase>) using parquet;


insert into l(a) values(array('a' collate utf8_binary_lcase));
insert into r(a) values(array('A' collate utf8_binary_lcase));


select * from l join r where l.a = r.a; 

Both runs should yield one row since the arrays are considered equal.

Issue Links

• links to: GitHub Pull Request #45611