Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6963

MR Map or Reduce specified node label is missing due to concurrent task limits

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.8.1
    • Fix Version/s: None
    • Component/s: mr-am
    • Labels:
      None

      Description

      In RMContainerAllocator#applyConcurrentTaskLimits, we limit the degree of task parallelism but not consider the node label specified for the ResourceRequest.

      private void applyConcurrentTaskLimits() {
          ....... 
            setRequestLimit(PRIORITY_FAST_FAIL_MAP, mapResourceRequest,
                failedMapRequestLimit);
            setRequestLimit(PRIORITY_MAP, mapResourceRequest, normalMapRequestLimit);
          }
          .......
            setRequestLimit(PRIORITY_REDUCE, reduceResourceRequest,
                reduceRequestLimit);
          }
        }
      

      Then we call the applyRequestLimits in RMContainerAllocator#makeRemoteRequest to apply the request limits. When the req.getNumContainers() > limit conditions are met,the original ResourceRequest(ask) will be replaced by the reqLimit which was generated by the above.

      private void applyRequestLimits() {
            .....
            // update an existing ask or send a new one if updating
            if (ask.remove(req) || requestLimitsToUpdate.contains(req)) {
              ResourceRequest newReq = req.getNumContainers() > limit
                  ? reqLimit : req;
              ask.add(newReq);
              LOG.info("Applying ask limit of " + newReq.getNumContainers()
                  + " for priority:" + reqLimit.getPriority()
                  + " and capability:" + reqLimit.getCapability());
            }
            .......
        }
      }
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              wei.wei wei
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: