[PIG-4329] Fetch optimization should be disabled when limit is not pushed up

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.15.0
• Assignee: Lorand Bendig
• Reporter: Cheolsoo Park
• Created: 13/Nov/14 05:46
• Updated: 07/Jun/15 03:47
• Resolved: 13/Nov/14 17:00

Description

Although PIG-4135 disable fetch optimization when there is no limit in the plan, that doesn't solve the problem completely. In fact, fetch optimization should be still disabled if limit is not pushed up. Consider the following query-

random_lists = load 'prodhive.schakraborty.search_server_denorm_impressions' using DseStorage();
random_lists = filter random_lists by entity_section=='random';
random_lists = limit random_lists 10;
dump random_lists;

Because the filter by blocks limit from being pushed up, POLoad actually scans the full table. In this case, fetch optimization makes the job extremely slow.