[PIG-4940] Predicate push-down filtering unary expressions can be pushed.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.17.0
• Hadoop Flags: Reviewed
• Assignee: Ryan Blue
• Reporter: Ryan Blue
• Created: 28/Jun/16 19:16
• Updated: 21/Jun/17 09:15
• Resolved: 19/Jul/16 00:49

Description

While testing predicate push-down, I ran into the following error:

Pig Exception

ERROR 0: Unsupported conversion of LogicalExpression to Expression: Map
        at org.apache.pig.newplan.FilterExtractor.getExpression(FilterExtractor.java:389)
        at org.apache.pig.newplan.FilterExtractor.getExpression(FilterExtractor.java:401)
        at org.apache.pig.newplan.FilterExtractor.getExpression(FilterExtractor.java:378)
        at org.apache.pig.newplan.FilterExtractor.getExpression(FilterExtractor.java:401)
        at org.apache.pig.newplan.FilterExtractor.getExpression(FilterExtractor.java:380)
        at org.apache.pig.newplan.FilterExtractor.visit(FilterExtractor.java:109)
        at org.apache.pig.newplan.PredicatePushDownFilterExtractor.visit(PredicatePushDownFilterExtractor.java:70)
        at org.apache.pig.newplan.logical.rules.PredicatePushdownOptimizer$PredicatePushDownTransformer.transform(PredicatePushdownOptimizer.java:146)
        at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
        ... 19 more

The problem is that the code is trying to push a map access operation, that isn't supported. The cause appears to be the logic in checkPushDown(UnaryExpression) that separates expressions that can be pushed from expressions that must be run by Pig. This function assumes that any expression under IsNullExpression or NotExpression can be pushed and adds the unary node's child expression to the pushdown expression without calling checkPushDown on it.

Issue Links

• relates to: PIG-4953 Predicate push-down will not run filters for single unary expressions
• links to: GitHub Pull Request #25
• links to: PR #25