[PIG-4240] Python UDF output serialization is invalid

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.12.0, 0.12.1, 0.13.0, 0.12.2, 0.13.1
• Assignee: Unassigned
• Reporter: Mike Sukmanowsky
• Created: 16/Oct/14 22:03
• Updated: 16/Oct/14 22:06

Description

The serialize_output function handles str and unicode return types from UDFs inappropriately.

Namely, a function could return a valid UTF-8 string, but the call to unicode will trigger a decode using the default Python encoding, ascii which can cause UnicodeDecodeError s.

Example:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# pig_test.py

s = "¼½¾"  # valid utf-8 string
print unicode(s, 'utf-8')  # works
print unicode(s)  # triggers UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128)

I propose making a few changes to serialize_output:

• Since ASCII (Python 2.x's default encoding) is a proper subset of UTF-8, there's no need to check for str being UTF-8 encoded, so we should remove checking where output_type == str
• Add a check for output_type == unicode that ensures utf-8 via output.encode('utf-8')

Patch forthcoming.