Pig
  1. Pig
  2. PIG-2681

TestDriverPig.countStores() does not correctly count the number of stores for pig scripts using variables for the alias

    Details

    • Type: Test Test
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.9.0, 0.9.1, 0.9.2, 0.10.0
    • Fix Version/s: 0.14.0
    • Component/s: e2e harness
    • Labels:
      None

      Description

      For pig macros where the out parameter is referenced in a store statement, the TestDriveP.countStores() does not correctly count the number of stores:

      For example, the store will not be counted in :

      define myMacro(in1,in2) returns A

      { A = load '$in1' using PigStorage('$delimeter') as (intnum1000: int,id: int,intnum5: int,intnum100: int,intnum: int,longnum: long,floatnum: float,doublenum: double); store $A into '$out'; }

      countStores() matches with:
      $count = $q[$i] =~ /store\s[a-zA-Z][a-zA-Z0-9_]*\s+into/i;

      Since the alias has a special character "$" it doesn't count it and the test fails.

      Need to change this to:

      $count = $q[$i] =~ /store\s(\$)?[a-zA-Z][a-zA-Z0-9_]*\s+into/i;

      I'll submit a patch shortly.

      1. PIG-2681.patch
        1 kB
        Araceli Henley

        Activity

        Araceli Henley created issue -
        Araceli Henley made changes -
        Field Original Value New Value
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.10.0 [ 12316246 ]
        Affects Version/s 0.9.2 [ 12318248 ]
        Affects Version/s 0.9.1 [ 12317343 ]
        Affects Version/s 0.9.0 [ 12315191 ]
        Fix Version/s 0.9.3 [ 12319456 ]
        Fix Version/s 0.11 [ 12318878 ]
        Fix Version/s 0.10.1 [ 12320547 ]
        Araceli Henley made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Araceli Henley made changes -
        Attachment PIG-2681.patch [ 12525689 ]
        Hide
        Araceli Henley added a comment -

        Includes test and resolution for pig harness

        Show
        Araceli Henley added a comment - Includes test and resolution for pig harness
        Araceli Henley made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Daniel Dai added a comment -

        +1, will commit soon.

        Show
        Daniel Dai added a comment - +1, will commit soon.
        Hide
        Daniel Dai added a comment -

        It is hard to use -r to expand Macro then do the counting? Otherwise if the Macro is not used (or used multiple times), we still get wrong counting.

        Show
        Daniel Dai added a comment - It is hard to use -r to expand Macro then do the counting? Otherwise if the Macro is not used (or used multiple times), we still get wrong counting.
        Hide
        Araceli Henley added a comment -

        True, it doesn't work for multiple invocations of the macro. I created a sepereate Jira for that issue: PIG-2682

        Show
        Araceli Henley added a comment - True, it doesn't work for multiple invocations of the macro. I created a sepereate Jira for that issue: PIG-2682
        Hide
        Daniel Dai added a comment -

        The fix only works if the macro is invoked once. Maybe a better and more elegant solution is needed.

        Show
        Daniel Dai added a comment - The fix only works if the macro is invoked once. Maybe a better and more elegant solution is needed.
        Hide
        Rohini Palaniswamy added a comment -

        Cancelling this patch as it is not a suitable solution for all cases. Easier solution would be to just have a variable that gives the number of stores for complex tests as suggested by Andrey Klochkov.

        Show
        Rohini Palaniswamy added a comment - Cancelling this patch as it is not a suitable solution for all cases. Easier solution would be to just have a variable that gives the number of stores for complex tests as suggested by Andrey Klochkov .
        Rohini Palaniswamy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Olga Natkovich made changes -
        Fix Version/s 0.12 [ 12323380 ]
        Fix Version/s 0.11 [ 12318878 ]
        Fix Version/s 0.9.3 [ 12319456 ]
        Fix Version/s 0.10.1 [ 12320547 ]
        Daniel Dai made changes -
        Fix Version/s 0.13.0 [ 12324971 ]
        Fix Version/s 0.12.0 [ 12323380 ]
        Aniket Mokashi made changes -
        Fix Version/s 0.14.0 [ 12326954 ]
        Fix Version/s 0.13.0 [ 12324971 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Araceli Henley
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development