[PIG-1669] PushUpFilter fail when filter condition contains scalar

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.8.0
• Component/s: impl
• Hadoop Flags: Reviewed
• Assignee: Daniel Dai
• Reporter: Daniel Dai
• Created: 05/Oct/10 21:05
• Updated: 30/Jun/17 21:26
• Resolved: 18/Oct/10 18:18

Description

The following script fail:

a = load 'studenttab10k' as (name, age, gpa);
b = group a all;
c = foreach b generate AVG(a.age) as age;
d = foreach a generate name, age;
e = filter d by age > c.age;
dump e;

Stack:
ERROR 2243: Attempt to remove operator LOFilter that is still softly connected in the plan

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias e
at org.apache.pig.PigServer.openIterator(PigServer.java:753)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:498)
at org.apache.pig.Main.main(Main.java:107)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias e
at org.apache.pig.PigServer.storeEx(PigServer.java:877)
at org.apache.pig.PigServer.store(PigServer.java:815)
at org.apache.pig.PigServer.openIterator(PigServer.java:727)
... 7 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2042: Error in new logical plan. Try -Dpig.usenewlogicalplan=false.
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:308)
at org.apache.pig.PigServer.compilePp(PigServer.java:1350)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1195)
at org.apache.pig.PigServer.storeEx(PigServer.java:873)
... 9 more
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule PushUpFilter. Try -t PushUpFilter
at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:120)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:276)
... 12 more
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2243: Attempt to remove operator LOFilter that is still softly connected in the plan
at org.apache.pig.newplan.BaseOperatorPlan.remove(BaseOperatorPlan.java:161)
at org.apache.pig.newplan.BaseOperatorPlan.removeAndReconnect(BaseOperatorPlan.java:423)
at org.apache.pig.newplan.logical.rules.FilterAboveForeach$FilterAboveForEachTransformer.transform(FilterAboveForeach.java:263)
at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)

Issue Links

• relates to: PIG-4767 Partition filter not pushed down when filter clause references variable from another load path