[TEZ-3708] Improve parallelism and auto grouping of unpartitioned cartesian product

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Assignee: Zhiyuan Yang
• Reporter: Zhiyuan Yang
• Created: 03/May/17 15:53
• Updated: 22/Aug/17 00:02
• Resolved: 12/May/17 00:57

Description

Current unpartitioned cartesian product has a few limitations
1. parallelism can be not enough in case of large split and small # src task
2. parallelism can be too much in in case of large # src task
3. workload is not ideally distributed across the worker. Even with auto grouping, grouping by size may not be accurate because same size can means different #record and different cartesian product ops.

Issue Links

• breaks: TEZ-3737 FairCartesianProductVertexMananger used incorrect #partition
• breaks: TEZ-3739 Fair CartesianProduct doesn't works well with huge difference in output size
• is blocked by: TEZ-3697 Adding #output_record in vertex manager event payload
• is related to: TEZ-3819 Round robin partitioner make fair cartesian product not fault tolerant