[MAPREDUCE-6963] MR Map or Reduce specified node label is missing due to concurrent task limits

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.8.1
• Component/s: mr-am
• Assignee: Unassigned
• Reporter: wei
• Created: 21/Sep/17 03:07
• Updated: 21/Sep/17 03:07

Description

In RMContainerAllocator#applyConcurrentTaskLimits, we limit the degree of task parallelism but not consider the node label specified for the ResourceRequest.

private void applyConcurrentTaskLimits() {
    ....... 
      setRequestLimit(PRIORITY_FAST_FAIL_MAP, mapResourceRequest,
          failedMapRequestLimit);
      setRequestLimit(PRIORITY_MAP, mapResourceRequest, normalMapRequestLimit);
    }
    .......
      setRequestLimit(PRIORITY_REDUCE, reduceResourceRequest,
          reduceRequestLimit);
    }
  }

Then we call the applyRequestLimits in RMContainerAllocator#makeRemoteRequest to apply the request limits. When the req.getNumContainers() > limit conditions are met，the original ResourceRequest(ask) will be replaced by the reqLimit which was generated by the above.

private void applyRequestLimits() {
      .....
      // update an existing ask or send a new one if updating
      if (ask.remove(req) || requestLimitsToUpdate.contains(req)) {
        ResourceRequest newReq = req.getNumContainers() > limit
            ? reqLimit : req;
        ask.add(newReq);
        LOG.info("Applying ask limit of " + newReq.getNumContainers()
            + " for priority:" + reqLimit.getPriority()
            + " and capability:" + reqLimit.getCapability());
      }
      .......
  }
}