[PIG-2316] Incorrect results for FILTER *** BY ( *** OR ***) with FilterLogicExpressionSimplifier optimizer turned on

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.8.0, 0.8.1, 0.9.0, 0.9.1
• Fix Version/s: 0.9.2, 0.10.0, 0.8.2

Release Note

FilterLogicExpressionSimplifier optimization rule is disabled by default. To enable the optimization rule, set the property pig.exec.filterLogicExpressionSimplifier to true.

• Assignee: Koji Noguchi
• Reporter: Huanyu Zhao
• Created: 08/Oct/11 00:58
• Updated: 18/Sep/13 04:10
• Resolved: 11/Oct/11 23:28

Description

An example for this bug:

cat weird.txt
1,a
2,b
3,c

When running pig with the following statements:

A = LOAD 'weird.txt' using PigStorage(',') AS (col1:int,col2);
B = FILTER A BY ((col1==1) OR (col1 != 1));
DUMP B;

I expect to get the result of all three rows back, but I receive only two rows.

(2,b)
(3,c)

When we start pig with optimizer turning off.

pig -optimizer_off All

With optimizer turning off, we get the expected results and I get three rows for the same statements.

(1,a)
(2,b)
(3,c)

--------------------------------------------------------

This bug was test on:

pig-0.9.1,
pig-0.9.0,
pig-0.8.1,
pig-0.8.0

All produced same incorrect results.

--------------------------------------------------------

When looked at the logical plan for this example, we found FilterlogicExpressionSimplifier optimizer produced incorrect logical plan. So we guess the bug is caused by FilterlogicExpressionSimplifier optimizer.

Issue Links

• relates to: PIG-3465 Fix problems with LogicalExpressionSimplifier and turn on the optimizer by default