IMPALA-2019 is resolved, we can document the UTF8_MODE query option added in it now. The query option will turn on the UTF-8 aware behavior of string functions. The relevant string functions and their UTF-8 aware behaviors are:
- LENGTH(STRING a)
- returns the number of UTF-8 characters instead of bytes
- SUBSTR(STRING a, INT start [, INT len])
SUBSTRING(STRING a, INT start [, INT len])()
- the substring start position and length is counted by UTF-8 characters instead of bytes
- REVERSE(STRING a)
- the unit of the operation is a UTF-8 character, ie. it won't reverse bytes inside a UTF-8 character.
- INSTR(STRING str, STRING substr[, BIGINT position[, BIGINT occurrence]])
LOCATE(STRING substr, STRING str[, INT pos])
- These functions have an optional position argument. The return values are also positions in the string. In UTF-8 mode, these positions are counted by UTF-8 characters instead of bytes.
- mask functions
- The unit of the operation is a UTF-8 character, ie. they won't mask the string byte-to-byte.
- These functions will recognize non-ascii characters and transform them based on the current locale used by the Impala process.