[PIG-4590] The function StringConcat returns a null value in certain cases even when there are chars to concatinate.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.12.0
• Component/s: grunt

Environment

Linux server

• Assignee: Unassigned
• Reporter: Jordan Babb
• Due: 10/Jun/15
• Created: 04/Jun/15 11:30
• Updated: 12/Jun/15 22:23

Description

The basic issue is I want to scan a whole row for a value. In the case below I want to take ref::tin and scan each column in combined or sub_load for that value.
Is there a better way to do this?
The way it is done below, a null value is returned for some of the rows when StringConcat is used. it seems to be if there is no value or null in the last column($21).
any help would be appreciated.

------------------------------------------

There should be 8 matches when files are combined.
When e is ran there should be 8 matches.

pig
ref_load = LOAD '/user/sditmpoc/pop/reference/TC4/RU99.KM40' USING PigStorage(',') AS (clnt_id:chararray, death_dt, birth_dt:chararray, tin:chararray, pref_first_name, cmprs_full_pref_first_name);
ref = FILTER ref_load BY (tin matches '^[^ ].*');

sub_load = LOAD '/user/sditmpoc/pop/Subject/TC4/RU99.KM40' using PigStorage(',') AS (clnt_id:chararray,two,three,four,birth_dt:chararray,six,tin:chararray,eight,nine,ten,eleven,twelve,thirteen,fourteen,fifteen,sixteen,seventeen,eighteen,nineteen,twenty,twentyone,twentytwo);
combined = JOIN sub_load BY (clnt_id), ref BY (clnt_id);
sub = foreach combined generate StringConcat($0,$1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21),CONCAT(CONCAT('.',ref::tin),'.');
e = filter sub by $0 matches $1;

example data for /user/sditmpoc/pop/reference/TC4/RU99.KM40:
BBBBB4WZ008,null,1933-07-28,402404671 ,,
BBBBB5WJ00P,null,1984-02-06, ,,
BBBBB6NHT04,null,1964-06-10, ,,

example data for /user/sditmpoc/pop/Subject/TC4/RU99.KM40:
BBBBB4WZ008,2013-08-12,F,null,1953-08-06,M, ,2,1,1,1,Y,null,N,N,402404671 ,2013-08-12 13:53:34.300944,2013-08-12 13:53:34.300993,N,4$,,a
BBBBB4WZ008,2013-08-12,F,null,1953-08-06,M, ,2,1,1,1,Y,null,N,N,402404671 ,2013-08-12 13:53:34.300944,2013-08-12 13:53:34.300993,N,4$,a,
BBBBB4WZ008,2013-08-12,F,null,1953-08-06,M, ,2,1,1,1,Y,null,N,N,402404671 ,2013-08-12 13:53:34.300944,2013-08-12 13:53:34.300993,N,4$,,
999986ZMYGF,2013-05-21,F,null,1989-12-21, , ,2,1,1,1,Y,2013-05-21,N,C,null,2013-05-21 14:11:33.300959,2013-06-23 09:27:20.608754,N,4$,,
BBBBB4WZ008,2013-08-27,M,null,1980-01-29,S, ,2,1,1,1,Y,2013-08-27,N,N,null,2013-08-27 14:12:09.30093,2013-08-27 14:12:09.300982,N,4$,,
9998975LHGF,2013-11-22,M,null,1963-10-20, , ,2,1,1,1,Y,2013-11-22,N,N,null,2013-11-22 13:55:49.300929,2013-11-22 13:55:49.300958,N,4$,,
BBBBB4WZ008,2013-08-12,F,null,1953-08-06,M, ,2,1,1,1,Y,null,N,N,402404671 ,2013-08-12 13:53:34.300944,2013-08-12 13:53:34.300993,N,4$,,
BBBBB4WZ008,2012-07-24,F,null,1972-01-07,M, ,2,28,1,1,Y,2012-07-24,N,N,null,2012-07-24 18:00:21.41841,2012-07-24 18:00:21.418635,N,04,402404671 ,
9998K71G9AL,2013-07-17, ,null,null, , ,2,1,1,1,Y,null,N,N,null,2013-07-17 13:59:51.301,2013-07-17 13:59:51.301082,N,4$,,
9998K6Y1TGF,2013-04-02, ,null,null, , ,3,1,1,1,Y,null,N,N,null,2013-04-02 14:00:07.30094,2013-04-02 14:00:07.301012,N,4$,,
9998K6YBQAL,2013-04-11, ,null,null, , ,2,1,1,1,Y,null,N,N,null,2013-04-11 14:00:04.301045,2013-04-11 14:00:04.301125,N,4$,,
9998H710HGF,2013-07-03, ,null,null, , ,2,1,1,1,Y,null,N,N,null,2013-07-03 13:58:55.301051,2013-07-03 13:58:55.301156,N,4$,,
BBBBB4WZ008,2012-08-07,M,null,1972-02-20,S, ,2,28,1,1,Y,2012-08-07,N,N,null,2012-08-07 17:58:40.055748,2012-08-07 17:58:40.055826,N,17,,
9998D6PWW0X,2012-08-23,F,null,1985-07-14,M, ,2,1,1,1,Y,2012-08-23,N,N,null,2012-08-23 17:57:36.472213,2012-08-23 17:57:36.472273,N,28,,
BBBBB4WZ008,2012-10-10,M,null,1980-04-16, , ,2,28,1,1,Y,2012-10-10,N,N,null,2012-10-10 18:50:12.545438,2012-10-10 18:50:12.545489,N,08,402404671 ,