[SPARK-47863] endsWith and startsWith don't work correctly for some collations

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Vladimir Golubev
• Reporter: Vladimir Golubev
• Created: 15/Apr/24 21:40
• Updated: 17/Apr/24 11:34
• Resolved: 17/Apr/24 11:34

Description

CollationSupport.EndsWIth and CollationSupport.StartsWith use CollationAwareUTF8String.matchAt, which operates byte offsets to compare prefixes/suffixes. This is not correct, since sometimes string parts (suffix/prefix) of different lengths are actually equal in context of case-insensitive and lower-case collations.

Example test cases that highlight the problem:

- assertContains("The İo", "i̇o", "UNICODE_CI", true); for CollationSupportSuite.testContains. 
- assertEndsWith("The İo", "i̇o", "UNICODE_CI", true); for CollationSupportSuite.testEndsWith.

The first passes, since it uses StringSearch directly, the second one does not.

Issue Links

• links to: GitHub Pull Request #46097