[PIG-3500] Initial implementation of TezCompiler

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: tez-branch
• Fix Version/s: tez-branch
• Component/s: tez
• Assignee: Cheolsoo Park
• Reporter: Cheolsoo Park
• Created: 04/Oct/13 00:24
• Updated: 21/Nov/14 05:58
• Resolved: 08/Oct/13 02:23

Description

Implement TezCompiler that compiles physical plan into tez plan. To begin with, we can implement the initial version that works for basic queries as follows:

1. Load-Filter-Store

   a = load 'file:///tmp/input' as (x:int, y:int);
   b = filter a by x > 0;
   c = foreach b generate y;
   store c into 'file:///tmp/output';
   

2. Load-Filter-GroupBy-Store

   a = load 'file:///tmp/input' as (x:int, y:int);
   b = group a by x;
   c = foreach b generate group, a;
   store c into 'file:///tmp/output';
   

3. Load1-Load2-Join-Store

   a = load 'file:///tmp/input1' as (x:int, y:int);
   b = load 'file:///tmp/input2' as (x:int, z:int);
   c = join a by x, b by x;
   d = foreach c generate a::x as x, y, z;
   store d into 'file:///tmp/output';
   

Issue Links

• is required by: PIG-3501 Initial implementation of TezJobControlCompiler
• is required by: PIG-3502 Initial implementation of TezLauncher