[TEZ-3452] Auto-reduce parallelism calculation can overflow with large inputs

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.2, 0.9.0, 0.8.5
• Assignee: Jonathan Turner Eagles
• Reporter: Jonathan Turner Eagles
• Created: 29/Sep/16 20:49
• Updated: 14/Mar/17 03:50
• Resolved: 20/Oct/16 16:18

Description

Overflow can occur when the numTasks is high (say 45000) and outputSize is high (say 311TB) and slow start is set to 1.0.

ShuffleVertexManager

    for (Map.Entry<String, SourceVertexInfo> vInfo : getBipartiteInfo()) {
      SourceVertexInfo srcInfo = vInfo.getValue();
      if (srcInfo.numTasks > 0 && srcInfo.numVMEventsReceived > 0) {
        // this assumes that 1 vmEvent is received per completed task - TEZ-2961
        expectedTotalSourceTasksOutputSize += 
            (srcInfo.numTasks * srcInfo.outputSize) / srcInfo.numVMEventsReceived;
      }
    }