[TEZ-3126] Log reason for not reducing parallelism

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.7.1, 0.8.3
• Assignee: Jonathan Turner Eagles
• Reporter: Jonathan Turner Eagles
• Created: 18/Feb/16 16:54
• Updated: 18/May/16 04:55
• Resolved: 23/Feb/16 04:50

Description

For example, when reducing parallelism from 36 to 22. The basePartitionRange will be 1 and will not re-configure the vertex.

ShuffleVertexManager#determineParallelismAndApply

    int desiredTaskParallelism = 
        (int)(
            (expectedTotalSourceTasksOutputSize+desiredTaskInputDataSize-1)/
            desiredTaskInputDataSize);
    if(desiredTaskParallelism < minTaskParallelism) {
      desiredTaskParallelism = minTaskParallelism;
    }
    
    if(desiredTaskParallelism >= currentParallelism) {
      return true;
    }
    
    // most shufflers will be assigned this range
    basePartitionRange = currentParallelism/desiredTaskParallelism;
    
    if (basePartitionRange <= 1) {
      // nothing to do if range is equal 1 partition. shuffler does it by default
      return true;
    }

Issue Links

• is related to: TEZ-2962 Use per partition stats in shuffle vertex manager auto parallelism