[YARN-2637] maximum-am-resource-percent could be respected for both LeafQueue/User when trying to activate applications.

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.7.0, 2.6.1, 3.0.0-alpha1
• Component/s: resourcemanager
• Labels: 2.6.1-candidate
• Hadoop Flags: Reviewed
• Assignee: Craig Welch
• Reporter: Wangda Tan
• Created: 01/Oct/14 22:23
• Updated: 25/Oct/19 20:26
• Resolved: 14/Jan/15 01:33

Description

Currently, number of AM in leaf queue will be calculated in following way:

max_am_resource = queue_max_capacity * maximum_am_resource_percent
#max_am_number = max_am_resource / minimum_allocation
#max_am_number_for_each_user = #max_am_number * userlimit * userlimit_factor

And when submit new application to RM, it will check if an app can be activated in following way:

    for (Iterator<FiCaSchedulerApp> i=pendingApplications.iterator(); 
         i.hasNext(); ) {
      FiCaSchedulerApp application = i.next();
      
      // Check queue limit
      if (getNumActiveApplications() >= getMaximumActiveApplications()) {
        break;
      }
      
      // Check user limit
      User user = getUser(application.getUser());
      if (user.getActiveApplications() < getMaximumActiveApplicationsPerUser()) {
        user.activateApplication();
        activeApplications.add(application);
        i.remove();
        LOG.info("Application " + application.getApplicationId() +
            " from user: " + application.getUser() + 
            " activated in queue: " + getQueueName());
      }
    }

An example is,
If a queue has capacity = 1G, max_am_resource_percent = 0.2, the maximum resource that AM can use is 200M, assuming minimum_allocation=1M, #am can be launched is 200, and if user uses 5M for each AM (> minimum_allocation). All apps can still be activated, and it will occupy all resource of a queue instead of only a max_am_resource_percent of a queue.

Issue Links

• breaks: YARN-5010 maxActiveApplications and maxActiveApplicationsPerUser are missing from REST API
• is related to: MAPREDUCE-6189 TestMRTimelineEventHandling fails in trunk