[PIG-1243] Passing Complex map types to and from streaming causes a problem

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.6.0
• Fix Version/s: 0.7.0
• Assignee: Richard Ding
• Reporter: Viraj Bhat
• Created: 18/Feb/10 23:26
• Updated: 02/May/13 02:29
• Resolved: 24/Feb/10 20:01

Description

I have a program which generates different types of Maps fields and stores it into PigStorage.

A = load '/user/viraj/three.txt' using PigStorage();

B = foreach A generate ['a'#'12'] as b:map[], ['b'#['c'#'12']] as c, ['c'#{(['d'#'15']),(['e'#'16'])}] as d;

store B into '/user/viraj/pigtest' using PigStorage();

Now I test the previous output in the below script to make sure I have the right results. I also pass this data to a Perl script and I observe that the complex Map types I have generated, are lost when I get the result back.

DEFINE CMD `simple.pl` SHIP('simple.pl');

A = load '/user/viraj/pigtest' using PigStorage() as (simpleFields, mapFields, mapListFields);

B = foreach A generate $0, $1, $2;

dump B;

C = foreach A generate  (chararray)simpleFields#'a' as value, $0,$1,$2;

D = stream C through CMD as (a0:map[], a1:map[], a2:map[]);

dump D;

dumping B results in:

(a#12,b#[c#12],[c#

{([d#15]),([e#16])}])
(a#12,b#[c#12],[c#{([d#15]),([e#16])}

])
(a#12,b#[c#12],[c#

{([d#15]),([e#16])}

])

dumping D results in:

(a#12,,)
(a#12,,)
(a#12,,)

The Perl script used here is:

#!/usr/local/bin/perl

use warnings;

use strict;

while(<>) {

    my($bc,$s,$m,$l)=split/\t/;

    print("$s\t$m\t$l");

}

Is there an issue with handling of complex Map fields within streaming? How can I fix this to obtain the right result?

Viraj

Issue Links

• depends upon: PIG-613 Casting complex type(tuple/bag/map) does not take effect