Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4590

The function StringConcat returns a null value in certain cases even when there are chars to concatinate.

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.12.0
    • None
    • grunt
    • None
    • Linux server

    Description

      The basic issue is I want to scan a whole row for a value. In the case below I want to take ref::tin and scan each column in combined or sub_load for that value.
      Is there a better way to do this?
      The way it is done below, a null value is returned for some of the rows when StringConcat is used. it seems to be if there is no value or null in the last column($21).
      any help would be appreciated.

      ------------------------------------------

      There should be 8 matches when files are combined.
      When e is ran there should be 8 matches.

      pig
      ref_load = LOAD '/user/sditmpoc/pop/reference/TC4/RU99.KM40' USING PigStorage(',') AS (clnt_id:chararray, death_dt, birth_dt:chararray, tin:chararray, pref_first_name, cmprs_full_pref_first_name);
      ref = FILTER ref_load BY (tin matches '^[^ ].*');

      sub_load = LOAD '/user/sditmpoc/pop/Subject/TC4/RU99.KM40' using PigStorage(',') AS (clnt_id:chararray,two,three,four,birth_dt:chararray,six,tin:chararray,eight,nine,ten,eleven,twelve,thirteen,fourteen,fifteen,sixteen,seventeen,eighteen,nineteen,twenty,twentyone,twentytwo);
      combined = JOIN sub_load BY (clnt_id), ref BY (clnt_id);
      sub = foreach combined generate StringConcat($0,$1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21),CONCAT(CONCAT('.',ref::tin),'.');
      e = filter sub by $0 matches $1;

      example data for /user/sditmpoc/pop/reference/TC4/RU99.KM40:
      BBBBB4WZ008,null,1933-07-28,402404671 ,,
      BBBBB5WJ00P,null,1984-02-06, ,,
      BBBBB6NHT04,null,1964-06-10, ,,

      example data for /user/sditmpoc/pop/Subject/TC4/RU99.KM40:
      BBBBB4WZ008,2013-08-12,F,null,1953-08-06,M, ,2,1,1,1,Y,null,N,N,402404671 ,2013-08-12 13:53:34.300944,2013-08-12 13:53:34.300993,N,4$,,a
      BBBBB4WZ008,2013-08-12,F,null,1953-08-06,M, ,2,1,1,1,Y,null,N,N,402404671 ,2013-08-12 13:53:34.300944,2013-08-12 13:53:34.300993,N,4$,a,
      BBBBB4WZ008,2013-08-12,F,null,1953-08-06,M, ,2,1,1,1,Y,null,N,N,402404671 ,2013-08-12 13:53:34.300944,2013-08-12 13:53:34.300993,N,4$,,
      999986ZMYGF,2013-05-21,F,null,1989-12-21, , ,2,1,1,1,Y,2013-05-21,N,C,null,2013-05-21 14:11:33.300959,2013-06-23 09:27:20.608754,N,4$,,
      BBBBB4WZ008,2013-08-27,M,null,1980-01-29,S, ,2,1,1,1,Y,2013-08-27,N,N,null,2013-08-27 14:12:09.30093,2013-08-27 14:12:09.300982,N,4$,,
      9998975LHGF,2013-11-22,M,null,1963-10-20, , ,2,1,1,1,Y,2013-11-22,N,N,null,2013-11-22 13:55:49.300929,2013-11-22 13:55:49.300958,N,4$,,
      BBBBB4WZ008,2013-08-12,F,null,1953-08-06,M, ,2,1,1,1,Y,null,N,N,402404671 ,2013-08-12 13:53:34.300944,2013-08-12 13:53:34.300993,N,4$,,
      BBBBB4WZ008,2012-07-24,F,null,1972-01-07,M, ,2,28,1,1,Y,2012-07-24,N,N,null,2012-07-24 18:00:21.41841,2012-07-24 18:00:21.418635,N,04,402404671 ,
      9998K71G9AL,2013-07-17, ,null,null, , ,2,1,1,1,Y,null,N,N,null,2013-07-17 13:59:51.301,2013-07-17 13:59:51.301082,N,4$,,
      9998K6Y1TGF,2013-04-02, ,null,null, , ,3,1,1,1,Y,null,N,N,null,2013-04-02 14:00:07.30094,2013-04-02 14:00:07.301012,N,4$,,
      9998K6YBQAL,2013-04-11, ,null,null, , ,2,1,1,1,Y,null,N,N,null,2013-04-11 14:00:04.301045,2013-04-11 14:00:04.301125,N,4$,,
      9998H710HGF,2013-07-03, ,null,null, , ,2,1,1,1,Y,null,N,N,null,2013-07-03 13:58:55.301051,2013-07-03 13:58:55.301156,N,4$,,
      BBBBB4WZ008,2012-08-07,M,null,1972-02-20,S, ,2,28,1,1,Y,2012-08-07,N,N,null,2012-08-07 17:58:40.055748,2012-08-07 17:58:40.055826,N,17,,
      9998D6PWW0X,2012-08-23,F,null,1985-07-14,M, ,2,1,1,1,Y,2012-08-23,N,N,null,2012-08-23 17:57:36.472213,2012-08-23 17:57:36.472273,N,28,,
      BBBBB4WZ008,2012-10-10,M,null,1980-04-16, , ,2,28,1,1,Y,2012-10-10,N,N,null,2012-10-10 18:50:12.545438,2012-10-10 18:50:12.545489,N,08,402404671 ,

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            ozil_11 Jordan Babb

            Dates

              Created:
              Updated:

              Slack

                Issue deployment