Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-5370

Union onschema + columnprune dropping used fields

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      After PIG-5312, below query started failing.

      A = load 'input.txt' as (a1:int, a2:chararray, a3:int);
      B = FOREACH (GROUP A by (a1,a2)) {
          A_FOREACH = FOREACH A GENERATE a2,a3;
          GENERATE A, FLATTEN(A_FOREACH) as (a2,a3);
      }
      C = load 'input2.txt' as (A:bag{tuple:(a1: int,a2: chararray,a3:int)},a2: chararray,a3:int);
      
      D = UNION ONSCHEMA B, C;
      
      dump D;
      
      input1.txt
      1       a       3
      2       b       4
      2       c       5
      1       a       6
      2       b       7
      1       c       8
      
      input2.txt
      {(10,a0,30),(20,b0,40)} zzz     222
      
      Expected output
      ({(10,a0,30),(20,b0,40)},zzz,222)
      ({(1,a,6),(1,a,3)},a,6)
      ({(1,a,6),(1,a,3)},a,3)
      ({(1,c,8)},c,8)
      ({(2,b,7),(2,b,4)},b,7)
      ({(2,b,7),(2,b,4)},b,4)
      ({(2,c,5)},c,5)
      
      Actual (incorrect) output
      ({(10,a0,30),(20,b0,40)})    ****ONLY 1 Field ****
      ({(1,a,6),(1,a,3)},a,6)
      ({(1,a,6),(1,a,3)},a,3)
      ({(1,c,8)},c,8)
      ({(2,b,7),(2,b,4)},b,7)
      ({(2,b,7),(2,b,4)},b,4)
      ({(2,c,5)},c,5)
      

        Attachments

        1. pig-5370-v1.patch
          5 kB
          Koji Noguchi
        2. pig-5370-v2.patch
          7 kB
          Koji Noguchi

          Issue Links

            Activity

              People

              • Assignee:
                knoguchi Koji Noguchi
                Reporter:
                knoguchi Koji Noguchi
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: