[SPARK-46837] String function support (parent)

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Spark Core
• Epic Link: Collation support
• Assignee: Unassigned
• Reporter: Aleksandar Tomic
• Created: 24/Jan/24 17:26
• Updated: 28/Apr/24 15:54

Description

TODO: List of all functions that need to be updated for collation support:

 

Feature/function	Priority	Type
Shuffle	0	comparison
Delta Columns	0	storage
Partition key	0	storage
Comparison operators	0	comparison
IN list	0	comparison
GROUP BY	0	comparison
MERGE, HASH joins	0	comparison
ORDER BY	0	sorting
Aggregation	0	comparison
like	0	comparison
regexp_*	0	matching
concat	0	Pass through
substr	0	Pass through
between	0	comparison
coalesce	0	Pass through
Is distinct	0	comparison
trim	0	Pass through
instr	0	comparison
lcase	0	Pass through, modify
lower	0	Pass through modify
replace	0	comparison
ucase	0	modify , pass through
upper	0	Modify, pass through
count(distinct )	0	comparison
min/max	0	Comparison pass through
array	0	Pass through
case	0	Pass through
decode	0	Pass through
elt	0	Comparison, pass through
Nullif, nvl, nvl2	0	Pass through, comparison
Session variables	1	storage
SQL UDF	1	Storage, pass through
Python UDF	1	Storage
Array element	1	storage
Map key	1	Storage, comparison
Map value	1	Storage
Struct field	1	storage
least/greatest	1	Comparison, pass through
if/iff/ifnull	1	Pass through, comparison
mapExpr [ keyExpr ]	1	Comparison, pass through
concat_ws	1	Pass through
contains	1	comparison
left	1	Pass through
*pad	1	Pass through
repeat	1	Pass through
reverse	1	Pass through
translate	1	Comparison, Pass through
array_agg	1	Pass through
first/last/any	1	Pass through
mode	1	Comparison, pass through
array_*	1	Pass through, dedup (array distinct)
explode	2	Pass through
filter	2	Pass through
flatten	2	Pass through
inline*	2	Pass through
reduce	2	Pass through
reverse	2	Pass through
shuffle	2	Pass through
Slice	2	Pass through
sort_array	2	Comparison, pass through
transform	2	Pass through
zip*	2	Pass through
map	2	Pass through
map_*	2	Pass through
str_to_map	2	Comparison, pass through
transform*	2	Pass through
stack	2	Pass through
describe	2	display
ilike	2	matching
charindex	2	comparison
endswith	2	comparison
startswith	2	comparison
find_in_set	2	comparison
initcap	2	Pass through, modify
locate	2	comparison
mask	2	Pass through
overlay	2	Pass through
position	2	comparison
sentences	2	Comparison, pass through
split	2	Comparison, pass through
split_part	2	Comparison, pass through
collect_list	2	Pass through
collect_set	2	Pass through
min_by/max_by	2	Comparison, pass through
Element_at, []	2	Pass through
aggregate	2	Pass through

Sub-Tasks

• contains, startswith, endswith (binary & lowercase collation only): Resolved
• contains (all collations): Resolved
• startswith, endswith (all collations): Resolved
• new test suite for UTF8String: Resolved
• fail all unsupported functions: Resolved
• Resolve AbstractDataType simpleStrings for StringTypeCollated: Resolved
• refactor UTF8String and CollationFactory: Resolved
• Fix CollationSupport test output: Resolved
• endsWith and startsWith don't work correctly for some collations: Resolved
• Add benchmark for stringpredicate expressions: Resolved
• Optimize string predicate expressions for UTF8_BINARY_LCASE collation: Resolved
• Regexp expressions (binary & lowercase collation only): Resolved
• Add support for ConcatWs & Elt (all collations): Resolved
• Add support for Upper, Lower, InitCap (all collations): Resolved
• Fix Upper, Lower, InitCap collation awareness: Resolved
• StringRepeat (all collations): Resolved
• StringReplace (all collations): Resolved
• Overlay, FormatString, Length, BitLength, OctetLength, SoundEx, Luhncheck (all collations): Resolved
• StringTranslate (all collations): Resolved
• StringTrim & StringTrimLeft/Right/Both (binary & lowercase collation only): Resolved
• StringInstr, FindInSet (all collations): Resolved
• StringLPad, StringRPad (all collations): Resolved
• Substring, Right, Left (all collations): Resolved
• Levenshtein (all collations): Open
• When the collationId is invalid, throw `COLLATION_INVALID_ID`: Open
• Ascii, Chr, Base64, UnBase64, Decode, StringDecode, Encode, ToBinary, FormatNumber, Sentences (all collations): Resolved
• SplitPart (binary & lowercase collation only): Resolved
• Add Collation Support for trim/ltrim/rtrim: Resolved
• Mode (all collations): Open
• StringToMap & Mask (all collations): Resolved
• Fix mathExpressions that use StringType: Resolved
• Use wildcard imports in CollationTypeCasts: Resolved
• Format expressions (all collations): Resolved
• Variant expressions (all collations): Resolved
• Add support for AbstractMapType: Resolved
• URL expressions (all collations): Resolved
• Miscellaneous expressions (all collations): Resolved
• CurrentLike - Database/Schema, Catalog, User (all collations): Open
• JSON expressions (all collations): Resolved
• CSV expressions (all collations): Resolved
• XML expressions (all collations): Resolved
• XPath expressions (all collations): Resolved
• inputFile expressions (all collations): Resolved
• DateFormatClass (all collations): Open
• Datetime expressions (all collations): Open
• Alter logic for: startsWith, endsWith, contains, locate (UTF8_BINARY_LCASE): Open
• Alter logic for: instr, substring_index (UTF8_BINARY_LCASE): Open
• Alter logic for: find_in_set, replace (UTF8_BINARY_LCASE): Open
• Implement modified Lowercase operation for UTF8_BINARY_LCASE: Open
• Add Expression Walker for Testing: Open