[TEZ-1163] Tez Auto Reducer-parallelism throws Divide-by-Zero

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.5.0
• Fix Version/s: 0.5.0
• Assignee: Rajesh Balamohan
• Reporter: Gopal Vijayaraghavan
• Created: 01/Jun/14 00:27
• Updated: 06/Sep/14 01:35
• Resolved: 02/Jul/14 04:45

Description

When the per-reducer estimate is set to 256Mb and over-all data output is ~200Mb, gets a divide-by-zero.

2014-05-31 16:17:29,125 FATAL [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
java.lang.ArithmeticException: / by zero
	at org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.determineParallelismAndApply(ShuffleVertexManager.java:392)
	at org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.schedulePendingTasks(ShuffleVertexManager.java:446)

Because of incorrect value set for minTasks

vertexmanager.ShuffleVertexManager: Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:true desiredTaskIput:256000000 minTasks:0

Issue Links

• is depended upon by: HIVE-7158 Use Tez auto-parallelism in Hive