[PIG-3846] Implement automatic reducer parallelism

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Component/s: tez
• Hadoop Flags: Reviewed
• Assignee: Daniel Dai
• Reporter: Rohini Palaniswamy
• Created: 27/Mar/14 22:55
• Updated: 21/Nov/14 05:59
• Resolved: 17/Jun/14 18:17

Description

Tez has it built-in. We can start with reusing it and then look at customization for better performance.

Issue Links

• is blocked by: TEZ-800 One-one edge with parallelism -1 fails if source vertex parallelism is not -1 as well
• is blocked by: TEZ-1143 1-1 source split event should be handled in Vertex.RUNNING and Vertex.INITED state
• is blocked by: TEZ-1145 Vertices should not start if they have uninitialized custom edges
• is blocked by: TEZ-1151 Vertex should stay in initializing state until custom vertex manager sets the parallelism
• is blocked by: TEZ-1164 Only events for tasks should be buffered in Initializing state
• is blocked by: TEZ-1171 Vertex remains in INITED state if all source vertices start while the vertex was in INITIALIZING state
• is depended upon by: PIG-4015 Provide a way to disable auto-parallism in tez
• relates to: TEZ-1170 Simplify Vertex Initializing transition