[IGNITE-14677] Calcite engine. Add 32-bit Unicode characters support

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Labels: calcite2-required calcite3-required
• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Unassigned
• Reporter: Aleksey Plekhanov
• Created: 04/May/21 07:31
• Updated: 12/Jul/21 19:55

Description

Currently, all SQL string length-dependent functions (length, substr, right, left, lpad, rpad, etc) work correctly only with up to 16-bit Unicode characters. These functions mapped to java String class functions (length, substring, etc), which operates with an array of 16-bit char's. For example, the result of {{"🦆".length()}} will be 2, and the result of SELECT CHAR_LENGTH('🦆') will be 2. 

Java String class has method offsetByCodePoints which can be used to implement functions that correctly work with 32-bit characters, but there will be a performance drop if this method will be used for all strings. Perhaps we need dedicated functions with 32-bit characters support.

Issue Links

• is duplicated by: IGNITE-15110 Calcite bug. Function CHAR_LENGTH works incorrect with UNICODE
• is part of: IGNITE-14535 Caclite SQL engine capabilities
• relates to: IGNITE-14545 Calcite engine. Unicode literal not supported