[PIG-4227] Streaming Python UDF handles bag outputs incorrectly

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Hadoop Flags: Reviewed
• Assignee: Cheolsoo Park
• Reporter: Cheolsoo Park
• Created: 10/Oct/14 01:06
• Updated: 21/Nov/14 05:59
• Resolved: 16/Oct/14 04:27

Description

I have a udf that generates different outputs when running as jython and streaming python.

jython

{([[BBC Worldwide]])}

streaming python

{(BC Worldwid)}

The problem is that streaming python encodes a bag output incorrectly. For this particular example, it serializes the output string as follows-

|{_[[BBC Worldwide]]|}_

where '|' and '_' wrap bag delimiters '{' and '}'. i.e. '{' => '|{_' and '}' => '|}_'.

But this is wrong because bag must contain tuples not chararrays. i.e. the correct encoding is as follows-

|{_|(_[[BBC Worldwide]]|)_|}_

where '|' and '_' wrap tuple delimiters '(' and ')' as well as bag delimiters.

This results in truncated outputs.