[PIG-4767] Partition filter not pushed down when filter clause references variable from another load path

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.15.0
• Fix Version/s: 0.18.0
• Hadoop Flags: Reviewed
• Assignee: Koji Noguchi
• Reporter: Anthony Hsu
• Created: 03/Jan/16 23:47
• Updated: 19/Jul/17 17:40
• Resolved: 13/Jul/17 18:29

Description

To reproduce:

test.pig

a = load 'a.txt';
a_group = group a all;
a_count = foreach a_group generate COUNT(a) as count;

b = load 'mytable' using org.apache.hcatalog.pig.HCatLoader();
b = filter b by datepartition == '2015-09-01-00' and foo == a_count.count;

dump b;

The above query ends up reading all the table partitions. If you remove the foo == a_count.count clause or replace a_count.count with a constant, then partition filtering happens properly.

Issue Links

• is related to: PIG-1669 PushUpFilter fail when filter condition contains scalar