Pig
  1. Pig
  2. PIG-2570

LimitOptimizer fails with dynamic LIMIT argument

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0, 0.11
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The following script fails with the exception shown below. Passing -t LimitOptimizer makes it work, as does adding an ORDER clause before the limit.

      A = LOAD 'data1.txt' AS (owner:chararray,pet:chararray,age:int,phone:chararray);
      B = group A all; 
      C = foreach B generate SUM(A.age) as total; 
      D = foreach A generate owner, age/(double)C.total AS percentAge; 
      F = LIMIT D C.total/8;
      DUMP F;
      
      Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule LimitOptimizer. Try -t LimitOptimizer
      	at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
      	at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:287)
      	at org.apache.pig.PigServer.compilePp(PigServer.java:1317)
      	at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1254)
      	at org.apache.pig.PigServer.storeEx(PigServer.java:953)
      	... 14 more
      Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2243: Attempt to remove operator LOLimit that is still softly connected in the plan
      	at org.apache.pig.newplan.BaseOperatorPlan.remove(BaseOperatorPlan.java:174)
      	at org.apache.pig.newplan.BaseOperatorPlan.removeAndReconnect(BaseOperatorPlan.java:449)
      	at org.apache.pig.newplan.logical.rules.LimitOptimizer$OptimizeLimitTransformer.transform(LimitOptimizer.java:124)
      	at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
      
      1. PIG-2570-2.patch
        4 kB
        Daniel Dai
      2. PIG-2570-1.patch
        3 kB
        Daniel Dai
      3. limit-fails.pig
        0.2 kB
        Bill Graham
      4. data1.txt
        0.1 kB
        Bill Graham

        Activity

        Bill Graham created issue -
        Hide
        Bill Graham added a comment -

        Attaching a sample script and datafile to reproduce with.

        Show
        Bill Graham added a comment - Attaching a sample script and datafile to reproduce with.
        Bill Graham made changes -
        Field Original Value New Value
        Attachment data1.txt [ 12517183 ]
        Attachment limit-fails.pig [ 12517184 ]
        Daniel Dai made changes -
        Attachment PIG-2570-1.patch [ 12517381 ]
        Daniel Dai made changes -
        Fix Version/s 0.10 [ 12316246 ]
        Fix Version/s 0.11 [ 12318878 ]
        Hide
        Bill Graham added a comment -

        +1

        Thanks Daniel! This patch does the trick.

        Show
        Bill Graham added a comment - +1 Thanks Daniel! This patch does the trick.
        Hide
        Thejas M Nair added a comment -

        It is possible to have multiple scalar variables in the limit expression, this will result in limit having multiple 'soft' predecessors. This case also needs to be handled in the patch.

        Show
        Thejas M Nair added a comment - It is possible to have multiple scalar variables in the limit expression, this will result in limit having multiple 'soft' predecessors. This case also needs to be handled in the patch.
        Hide
        Daniel Dai added a comment -

        Good catch. Attach PIG-2570-2.patch.

        Show
        Daniel Dai added a comment - Good catch. Attach PIG-2570 -2.patch.
        Daniel Dai made changes -
        Attachment PIG-2570-2.patch [ 12518222 ]
        Hide
        Dmitriy V. Ryaboy added a comment -

        +1, please commit.

        Show
        Dmitriy V. Ryaboy added a comment - +1, please commit.
        Hide
        Daniel Dai added a comment -

        test-patch:
        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] -1 release audit. The applied patch generated 535 release audit warnings (more than the trunk's current 532 warnings).

        javadoc and release audit warning is unrelated.

        Patch committed to 0.10/trunk.

        Show
        Daniel Dai added a comment - test-patch: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 535 release audit warnings (more than the trunk's current 532 warnings). javadoc and release audit warning is unrelated. Patch committed to 0.10/trunk.
        Daniel Dai made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Assignee Bill Graham [ billgraham ] Daniel Dai [ daijy ]
        Resolution Fixed [ 1 ]
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        7d 21h 31m 1 Daniel Dai 13/Mar/12 23:22
        Resolved Resolved Closed Closed
        43d 21h 10m 1 Daniel Dai 26/Apr/12 21:33

          People

          • Assignee:
            Daniel Dai
            Reporter:
            Bill Graham
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development