[PIG-3634] Improve performance of order-by

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: tez-branch
• Component/s: tez
• Assignee: Daniel Dai
• Reporter: Daniel Dai
• Created: 18/Dec/13 19:33
• Updated: 21/Nov/14 05:59
• Resolved: 01/Jan/14 00:08

Description

This is a followup for PIG-3534. In PIG-3534, we use 5 vertexes (3 DAGs) to implement an order-by. We can optimize to use 4 vertexes in 1 DAG:

vertex 1: close the current vertex, create input + samples input
vertex 2: aggregate samples to create quantiles
vertex 3: use quantiles to partition input
vertex 4: sort input after partition

The DAG is:

vertex 1   ------------------>  vertex 3 ------> vertex 4
           \--> vertex 2 ---/

Issue Links

• depends upon: TEZ-683 Diamond shape DAG fail