[IMPALA-9205] UDF function in impala recieved chinese character change to???

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: Impala 2.12.0
• Environment: CentOS 7.3 ; Hive 1.2 ; Impala 2.12 ; Java JDK 1.8 ; Python 2.7.5
• Epic Color: ghx-label-10
• Assignee: Unassigned
• Reporter: Moxuan Shi
• Created: 28/Nov/19 09:46
• Updated: 02/Dec/19 03:10
• Resolved: 01/Dec/19 01:43

Description

UDF works in hive, but not in impala.

 

select leftcutcontentudf("一二三",2);

OK
一二

work in hive

 

select leftcutcontentudf("一二三",2);

+----------------------------------------+
| default.leftcutcontentudf('一二三', 2) |
+----------------------------------------+
| ??                                     |
+----------------------------------------+

chinese character changed to ?? in impala

 

I make a new UDF to print byte for input String

public class GetBytes extends UDF {

    public String evaluate(String input) {
        byte[] bytes = input.getBytes();
        StringBuffer stringBuffer = new StringBuffer();
        for (byte b : bytes){
            stringBuffer.append(b).append(" ");
        }
        return stringBuffer.toString();
    }
}

it seems that the chinese character changed to ??? before calling UDF function.

select getbytes("一二三");

+-----------------------------+ 
| default.getbytes('一二三') | 
+-----------------------------+ 
| 63 63 63 63 63 63 63 63 63 | 
+-----------------------------+

GetBytes result

 

but normal query is correct in impala.

select khmc_62c57e8ae0ac from collective_2085;

+-------------------+
| khmc_62c57e8ae0ac |
+-------------------+
| 淘宝              |
+-------------------+

correct query

how to deal with this problem?

Issue Links

• duplicates: IMPALA-2019 Proper UTF-8 support in string functions