[THRIFT-4677] UnicodeDecodeError in Python3

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Python - Library

Environment

Operating System: Windows 10 Pro (Simplified Chinese)

Python Interpreter: Python 3.6.6

osquery Version: 3.3.0

osquery-python Version: 3.0.5

 

• Assignee: Unassigned
• Reporter: Jarry Shaw
• Created: 30/Nov/18 13:41
• Updated: 12/Aug/19 15:37

Description

This is an issue occurred when using osquery-python (Python binding of osquery by Facebook).

When querying, UnicodeDecodeError raised with error message: "'utf-8' codec can't decode byte 0xc3 in position 0: invalid continuation byte" from thrift.compat.binary_to_str, which is because the encoding of bin_val parameter should be "gbk".

Possible approaches are:

• add a parameter for user to determine encodings
• get the system encoding through locale.getpreferredencoding()
• call bin_val.decode with errors='replace' or errors='ignore' parameter
• introduce chardet to try and resolve encoding problems

The attachment is my hack solution to this issue (through not perfect).