Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2570

LimitOptimizer fails with dynamic LIMIT argument

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0, 0.11
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The following script fails with the exception shown below. Passing -t LimitOptimizer makes it work, as does adding an ORDER clause before the limit.

      A = LOAD 'data1.txt' AS (owner:chararray,pet:chararray,age:int,phone:chararray);
      B = group A all; 
      C = foreach B generate SUM(A.age) as total; 
      D = foreach A generate owner, age/(double)C.total AS percentAge; 
      F = LIMIT D C.total/8;
      DUMP F;
      
      Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule LimitOptimizer. Try -t LimitOptimizer
      	at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
      	at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:287)
      	at org.apache.pig.PigServer.compilePp(PigServer.java:1317)
      	at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1254)
      	at org.apache.pig.PigServer.storeEx(PigServer.java:953)
      	... 14 more
      Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2243: Attempt to remove operator LOLimit that is still softly connected in the plan
      	at org.apache.pig.newplan.BaseOperatorPlan.remove(BaseOperatorPlan.java:174)
      	at org.apache.pig.newplan.BaseOperatorPlan.removeAndReconnect(BaseOperatorPlan.java:449)
      	at org.apache.pig.newplan.logical.rules.LimitOptimizer$OptimizeLimitTransformer.transform(LimitOptimizer.java:124)
      	at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
      
      1. data1.txt
        0.1 kB
        Bill Graham
      2. limit-fails.pig
        0.2 kB
        Bill Graham
      3. PIG-2570-1.patch
        3 kB
        Daniel Dai
      4. PIG-2570-2.patch
        4 kB
        Daniel Dai

        Activity

        Hide
        daijy Daniel Dai added a comment -

        test-patch:
        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] -1 release audit. The applied patch generated 535 release audit warnings (more than the trunk's current 532 warnings).

        javadoc and release audit warning is unrelated.

        Patch committed to 0.10/trunk.

        Show
        daijy Daniel Dai added a comment - test-patch: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 535 release audit warnings (more than the trunk's current 532 warnings). javadoc and release audit warning is unrelated. Patch committed to 0.10/trunk.
        Hide
        dvryaboy Dmitriy V. Ryaboy added a comment -

        +1, please commit.

        Show
        dvryaboy Dmitriy V. Ryaboy added a comment - +1, please commit.
        Hide
        daijy Daniel Dai added a comment -

        Good catch. Attach PIG-2570-2.patch.

        Show
        daijy Daniel Dai added a comment - Good catch. Attach PIG-2570 -2.patch.
        Hide
        thejas Thejas M Nair added a comment -

        It is possible to have multiple scalar variables in the limit expression, this will result in limit having multiple 'soft' predecessors. This case also needs to be handled in the patch.

        Show
        thejas Thejas M Nair added a comment - It is possible to have multiple scalar variables in the limit expression, this will result in limit having multiple 'soft' predecessors. This case also needs to be handled in the patch.
        Hide
        billgraham Bill Graham added a comment -

        +1

        Thanks Daniel! This patch does the trick.

        Show
        billgraham Bill Graham added a comment - +1 Thanks Daniel! This patch does the trick.
        Hide
        billgraham Bill Graham added a comment -

        Attaching a sample script and datafile to reproduce with.

        Show
        billgraham Bill Graham added a comment - Attaching a sample script and datafile to reproduce with.

          People

          • Assignee:
            daijy Daniel Dai
            Reporter:
            billgraham Bill Graham
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development