[SPARK-46837] String function support (parent)

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Epic Link: Collation support
• Assignee: Unassigned
• Reporter: Aleksandar Tomic
• Created: 24/Jan/24 17:26
• Updated: 01/Oct/24 17:48

Description

TODO: List of all functions that need to be updated for collation support:

 

Feature/function	Priority	Type
Shuffle	0	comparison
Delta Columns	0	storage
Partition key	0	storage
Comparison operators	0	comparison
IN list	0	comparison
GROUP BY	0	comparison
MERGE, HASH joins	0	comparison
ORDER BY	0	sorting
Aggregation	0	comparison
like	0	comparison
regexp_*	0	matching
concat	0	Pass through
substr	0	Pass through
between	0	comparison
coalesce	0	Pass through
Is distinct	0	comparison
trim	0	Pass through
instr	0	comparison
lcase	0	Pass through, modify
lower	0	Pass through modify
replace	0	comparison
ucase	0	modify , pass through
upper	0	Modify, pass through
count(distinct )	0	comparison
min/max	0	Comparison pass through
array	0	Pass through
case	0	Pass through
decode	0	Pass through
elt	0	Comparison, pass through
Nullif, nvl, nvl2	0	Pass through, comparison
Session variables	1	storage
SQL UDF	1	Storage, pass through
Python UDF	1	Storage
Array element	1	storage
Map key	1	Storage, comparison
Map value	1	Storage
Struct field	1	storage
least/greatest	1	Comparison, pass through
if/iff/ifnull	1	Pass through, comparison
mapExpr [ keyExpr ]	1	Comparison, pass through
concat_ws	1	Pass through
contains	1	comparison
left	1	Pass through
*pad	1	Pass through
repeat	1	Pass through
reverse	1	Pass through
translate	1	Comparison, Pass through
array_agg	1	Pass through
first/last/any	1	Pass through
mode	1	Comparison, pass through
array_*	1	Pass through, dedup (array distinct)
explode	2	Pass through
filter	2	Pass through
flatten	2	Pass through
inline*	2	Pass through
reduce	2	Pass through
reverse	2	Pass through
shuffle	2	Pass through
Slice	2	Pass through
sort_array	2	Comparison, pass through
transform	2	Pass through
zip*	2	Pass through
map	2	Pass through
map_*	2	Pass through
str_to_map	2	Comparison, pass through
transform*	2	Pass through
stack	2	Pass through
describe	2	display
ilike	2	matching
charindex	2	comparison
endswith	2	comparison
startswith	2	comparison
find_in_set	2	comparison
initcap	2	Pass through, modify
locate	2	comparison
mask	2	Pass through
overlay	2	Pass through
position	2	comparison
sentences	2	Comparison, pass through
split	2	Comparison, pass through
split_part	2	Comparison, pass through
collect_list	2	Pass through
collect_set	2	Pass through
min_by/max_by	2	Comparison, pass through
Element_at, []	2	Pass through
aggregate	2	Pass through

Issue Links

• is related to: SPARK-44111 Prepare Apache Spark 4.0.0
• links to: GitHub Pull Request #47154

Sub-Tasks

• PandasMode (all collations): Open
• Fix SchemaOfJson Expression to work with Collations: Open
• Expand collation benchmark coverage: Open
• Support collations with get_json_object and json_tuple: Open