Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
A = load ...
B1 = filter A by cond1;
B2 = filter A by cond2;
B3 = filter A by cond3;
B = union B1, B2, B3; does not push projection.
Load (A)
Split
---------------------------
Filter(B1) Filter(B2) ...
Because of the split operator introduced between the filter conditions and load, the filter does not get pushed into the load function.
A simple way to fix this in pig would be to not share the load across the filter operators. Another option is to push the condition (B1 or B2 or B3) into Load operator and retain rest of the current plan (split and filters following the split).
You can of course achieve the same effect by having a separate load statement as input for each of the filters.