Pig
  1. Pig
  2. PIG-2570

LimitOptimizer fails with dynamic LIMIT argument

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0, 0.11
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The following script fails with the exception shown below. Passing -t LimitOptimizer makes it work, as does adding an ORDER clause before the limit.

      A = LOAD 'data1.txt' AS (owner:chararray,pet:chararray,age:int,phone:chararray);
      B = group A all; 
      C = foreach B generate SUM(A.age) as total; 
      D = foreach A generate owner, age/(double)C.total AS percentAge; 
      F = LIMIT D C.total/8;
      DUMP F;
      
      Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule LimitOptimizer. Try -t LimitOptimizer
      	at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
      	at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:287)
      	at org.apache.pig.PigServer.compilePp(PigServer.java:1317)
      	at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1254)
      	at org.apache.pig.PigServer.storeEx(PigServer.java:953)
      	... 14 more
      Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2243: Attempt to remove operator LOLimit that is still softly connected in the plan
      	at org.apache.pig.newplan.BaseOperatorPlan.remove(BaseOperatorPlan.java:174)
      	at org.apache.pig.newplan.BaseOperatorPlan.removeAndReconnect(BaseOperatorPlan.java:449)
      	at org.apache.pig.newplan.logical.rules.LimitOptimizer$OptimizeLimitTransformer.transform(LimitOptimizer.java:124)
      	at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
      
      1. PIG-2570-2.patch
        4 kB
        Daniel Dai
      2. PIG-2570-1.patch
        3 kB
        Daniel Dai
      3. limit-fails.pig
        0.2 kB
        Bill Graham
      4. data1.txt
        0.1 kB
        Bill Graham

        Activity

        Hide
        Bill Graham added a comment -

        Attaching a sample script and datafile to reproduce with.

        Show
        Bill Graham added a comment - Attaching a sample script and datafile to reproduce with.
        Hide
        Bill Graham added a comment -

        +1

        Thanks Daniel! This patch does the trick.

        Show
        Bill Graham added a comment - +1 Thanks Daniel! This patch does the trick.
        Hide
        Thejas M Nair added a comment -

        It is possible to have multiple scalar variables in the limit expression, this will result in limit having multiple 'soft' predecessors. This case also needs to be handled in the patch.

        Show
        Thejas M Nair added a comment - It is possible to have multiple scalar variables in the limit expression, this will result in limit having multiple 'soft' predecessors. This case also needs to be handled in the patch.
        Hide
        Daniel Dai added a comment -

        Good catch. Attach PIG-2570-2.patch.

        Show
        Daniel Dai added a comment - Good catch. Attach PIG-2570 -2.patch.
        Hide
        Dmitriy V. Ryaboy added a comment -

        +1, please commit.

        Show
        Dmitriy V. Ryaboy added a comment - +1, please commit.
        Hide
        Daniel Dai added a comment -

        test-patch:
        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] -1 release audit. The applied patch generated 535 release audit warnings (more than the trunk's current 532 warnings).

        javadoc and release audit warning is unrelated.

        Patch committed to 0.10/trunk.

        Show
        Daniel Dai added a comment - test-patch: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 535 release audit warnings (more than the trunk's current 532 warnings). javadoc and release audit warning is unrelated. Patch committed to 0.10/trunk.

          People

          • Assignee:
            Daniel Dai
            Reporter:
            Bill Graham
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development