[TEZ-1248] Reduce slow-start should special case 1 reducer runs

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.5.0
• Fix Version/s: 0.9.0, 0.8.5

Environment

20 node cluster running tez

• Target Version/s: 0.8.5
• Assignee: Zhiyuan Yang
• Reporter: Gopal Vijayaraghavan
• Created: 01/Jul/14 19:46
• Updated: 14/Mar/17 03:49
• Resolved: 11/Jul/16 23:57

Description

Reducer slow-start has a performance problem for the small cases where there is just 1 reducer for a case with a single wave.

Tez knows the split count and wave count, being able to determine if the cluster has enough spare capacity to run the reducer earlier for lower latency in a N-mapper -> 1 reducer case.

Issue Links

• relates to: HIVE-14200 Tez: disable auto-reducer parallelism when reducer-count * min.partition.factor < 1.0