[IMPALA-11127] Document the UTF8_MODE query option and relavent string functions

• Type: Documentation
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 4.1.0
• Fix Version/s: Impala 4.1.0
• Target Version: Impala 4.1.0
• Epic Color: ghx-label-6
• Assignee: shajini thayasingh
• Reporter: Quanlong Huang
• Created: 16/Feb/22 00:45
• Updated: 22/Apr/22 02:29
• Resolved: 22/Apr/22 02:29

Description

Since IMPALA-2019 is resolved, we can document the UTF8_MODE query option added in it now. The query option will turn on the UTF-8 aware behavior of string functions. The relevant string functions and their UTF-8 aware behaviors are:

• LENGTH(STRING a)
  • returns the number of UTF-8 characters instead of bytes
• SUBSTR(STRING a, INT start [, INT len])
  SUBSTRING(STRING a, INT start [, INT len])()
  • the substring start position and length is counted by UTF-8 characters instead of bytes
• REVERSE(STRING a)
  • the unit of the operation is a UTF-8 character, ie. it won't reverse bytes inside a UTF-8 character.
• INSTR(STRING str, STRING substr[, BIGINT position[, BIGINT occurrence]])
  LOCATE(STRING substr, STRING str[, INT pos])
  • These functions have an optional position argument. The return values are also positions in the string. In UTF-8 mode, these positions are counted by UTF-8 characters instead of bytes.
• mask functions
  • The unit of the operation is a UTF-8 character, ie. they won't mask the string byte-to-byte.
• upper/lower/initcap
  • These functions will recognize non-ascii characters and transform them based on the current locale used by the Impala process.