[TEZ-2940] Invalid shuffle max slow start setting causes vertex to hang indefinitely

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.1, 0.8.2
• Target Version/s: 0.7.1
• Assignee: Jonathan Turner Eagles
• Reporter: Jonathan Turner Eagles
• Created: 12/Nov/15 17:29
• Updated: 18/May/16 04:54
• Resolved: 12/Nov/15 19:58

Description

If invalid shuffle vertex manager max slow start setting is used, vertex can defer indefinitely if output completed tasks output size is less than the desired input task size.

2015-11-12 14:54:48,378 [INFO] [Dispatcher thread {Central}] |vertexmanager.ShuffleVertexManager|: Shuffle Vertex Manager: settings minFrac:0.9 maxFrac:95.0 auto:true desiredTaskIput:134217728 minTasks:1

2015-11-12 14:55:31,432 [INFO] [App Shared Pool - #6] |vertexmanager.ShuffleVertexManager|: Defer scheduling tasks; vertex=scope-218, totalNumBipartiteSourceTasks=165, completedSourceTasksOutputSize=977373325, numVertexManagerEventsReceived=165, numBipartiteSourceTasksCompleted=165, maxThreshold=15675.0