[IMPALA-2019] Proper UTF-8 support in string functions

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.1, Impala 2.2
• Fix Version/s: Impala 4.1.0
• Component/s: Backend
• Labels: sql-language
• Epic Link: UTF-8
• Target Version: Product Backlog
• Assignee: Quanlong Huang
• Reporter: Andrés Cordero
• Created: 19/May/15 18:34
• Updated: 16/Jan/24 08:19
• Resolved: 15/Feb/22 23:47

Description

As documented here: https://impala.apache.org/docs/build/html/topics/impala_string.html
Impala does not properly handle non-ASCII UTF-8 characters, and will return results in string functions such as length that are inconsistent with Hive.

Issue Links

• Blocked: IMPALA-7511 translate() string function only works with ascii characters.
• is blocked by: IMPALA-10416 Testfile can't deal with non-ascii results
• is duplicated by: IMPALA-9205 UDF function in impala recieved chinese character change to???
• is related to: IMPALA-12718 trim() functions are lack of utf-8 support
• is related to: IMPALA-4823 Create table with multi-byte column fails with AnalysisException
• is related to: IMPALA-9662 Add builtin functions for masking UTF-8 strings
• is related to: HIVE-18083 Support UTF8 in MySQL Metastore Backend
• is related to: IMPALA-11519 Document about UTF-8 support requirements
• relates to: IMPALA-6653 Unicode support for Kudu table names
• relates to: IMPALA-5675 Support CHAR/VARCHAR length counted in number of UTF-8 characters, not bytes
• relates to: IMPALA-4290 Order by appears to produce incorrect result for strings with mixed English and Chinese characters
• relates to: IMPALA-792 string functions does not support multibyte character set (MBCS)
• relates to: IMPALA-3478 Support for UTF-8 BOM on text backed tables.