[PIG-1926] Sample/Limit should take scalar

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Labels: gsoc2011

Release Note

Limit and Sample now accept a variable (scalar) as argument.

For example, the new Limit command allows the following syntax to get the top 1% of a sorted file:
[ a = LOAD 'a.txt'; b = GROUP a all; c = FOREACH b GENERATE COUNT(a) AS sum; d = ORDER a BY $0; e = LIMIT d c.sum/100; ]

Only scalar variables may be used in the expression in limit or sample, columns in the input relation for the operation cannot be used in the expression. A statement like [ e = LIMIT d $0; ] is invalid.
The new Sample command allows for the same syntax.

Using a variable instead of a constant in Limit automatically disables most of the optimizations (only push-before-foreach is performed). More work is needed to enable optimizations for limit-after-sort, limit duplication before cross/union and limit merging.

• Assignee: Gianmarco De Francisci Morales
• Reporter: Daniel Dai
• Created: 21/Mar/11 19:26
• Updated: 26/Apr/12 20:32
• Resolved: 07/Jul/11 00:30

Description

Currently, Limit, Sample only takes a constant. It would be better we can use a scalar in the place of constant. Eg:

a = load 'a.txt';
b = group a all;
c = foreach b generate COUNT(a) as sum;
d = order a by $0;
e = limit d c.sum/100;

This is a candidate project for Google summer of code 2011. More information about the program can be found at http://wiki.apache.org/pig/GSoc2011

Issue Links

• relates to: PIG-2156 Limit/Sample with variable does not work if the expression starts with an integer/double