[PIG-1321] Logical Optimizer: Merge cascading foreach - ASF JIRA

XML

Word

Printable

JSON

Hadoop Flags:

Reviewed
Release Note:

Hide
This rule allows to merge togther two foreach statements if the following preconditions are met:

- foreach statements are consecutive
- the second foreach is not nested
- the first foreach statement does not contain flatten

Example:

(1) Original code:

A = load 'file.txt' as (a, b, c);
B = foreach A generate a+b as u, c-b as v;
C = foreach B generate $0+5, v;
.....

(2) Optimized code:

A = load 'file.txt' as (a, b, c);
C = foreach A generate a+b+5, c-b;
......

Show
This rule allows to merge togther two foreach statements if the following preconditions are met: - foreach statements are consecutive - the second foreach is not nested - the first foreach statement does not contain flatten Example: (1) Original code: A = load 'file.txt' as (a, b, c); B = foreach A generate a+b as u, c-b as v; C = foreach B generate $0+5, v; ..... (2) Optimized code: A = load 'file.txt' as (a, b, c); C = foreach A generate a+b+5, c-b; ......

We can merge consecutive foreach statement.

Eg:
b = foreach a generate a0#'key1' as b0, a0#'key2' as b1, a1;
c = foreach b generate b0#'kk1', b0#'kk2', b1, a1;

=> c = foreach a generate a0#'key1'#'kk1', a0#'key1'#'kk2', a0#'key2', a1;