Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-910

PushUpFilter: Filter after skew LOJoin generates an error

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Invalid
    • 0.3.0
    • 0.4.0
    • impl
    • None

    Description

      The script is:
      A = load 'a' as (user:chararray, action, timespent, query_term, ip_addr, timestamp, estimated_revenue, page_info, page_links);
      alpha = load 'b' as (user:chararray, action, timespent, query_term, ip_addr, timestamp, estimated_revenue, page_info, page_links);
      C = join A by user, alpha by user using "skewed" parallel 40;
      D = filter C by A::user == 'aaa';
      store D into 'L5out';

      Stack trace is:
      ERROR 1093: Swap supports swap of operators with at most one input. Found first operator with 2 inputs.
      org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2151:
      Internal error while pushing filters up.
      at org.apache.pig.impl.logicalLayer.optimizer.PushUpFilter.transform(PushUpFilter.java: 325)
      at org.apache.pig.impl.logicalLayer.optimizer.LogicalOptimizer.optimize(LogicalOptimizer.java:185)
      at org.apache.pig.PigServer.compileLp(PigServer.java:854)
      at org.apache.pig.PigServer.compileLp(PigServer.java:791)
      at org.apache.pig.PigServer.execute(PigServer.java:760)
      at org.apache.pig.PigServer.access$100(PigServer.java:95)
      at org.apache.pig.PigServer$Graph.execute(PigServer.java:943)
      at org.apache.pig.PigServer.executeBatch(PigServer.java:255)
      at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java: 112)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java: 169)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java: 141)
      at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
      at org.apache.pig.Main.main(Main.java:320)
      Caused by: org.apache.pig.impl.plan.PlanException: ERROR 1093: Swap supports swap of operators with at most one input. Found first operator with 2 inputs.
      at org.apache.pig.impl.plan.OperatorPlan.swap(OperatorPlan.java:964)
      at org.apache.pig.impl.logicalLayer.optimizer.PushUpFilter.transform(PushUpFilter.java: 310)
      ... 12 more

      Attachments

        1. PIG-910-1.patch
          4 kB
          Daniel Dai

        Activity

          People

            Unassigned Unassigned
            daijy Daniel Dai
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: