Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2637

maximum-am-resource-percent could be respected for both LeafQueue/User when trying to activate applications.

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Currently, number of AM in leaf queue will be calculated in following way:

      max_am_resource = queue_max_capacity * maximum_am_resource_percent
      #max_am_number = max_am_resource / minimum_allocation
      #max_am_number_for_each_user = #max_am_number * userlimit * userlimit_factor
      

      And when submit new application to RM, it will check if an app can be activated in following way:

          for (Iterator<FiCaSchedulerApp> i=pendingApplications.iterator(); 
               i.hasNext(); ) {
            FiCaSchedulerApp application = i.next();
            
            // Check queue limit
            if (getNumActiveApplications() >= getMaximumActiveApplications()) {
              break;
            }
            
            // Check user limit
            User user = getUser(application.getUser());
            if (user.getActiveApplications() < getMaximumActiveApplicationsPerUser()) {
              user.activateApplication();
              activeApplications.add(application);
              i.remove();
              LOG.info("Application " + application.getApplicationId() +
                  " from user: " + application.getUser() + 
                  " activated in queue: " + getQueueName());
            }
          }
      

      An example is,
      If a queue has capacity = 1G, max_am_resource_percent = 0.2, the maximum resource that AM can use is 200M, assuming minimum_allocation=1M, #am can be launched is 200, and if user uses 5M for each AM (> minimum_allocation). All apps can still be activated, and it will occupy all resource of a queue instead of only a max_am_resource_percent of a queue.

        Attachments

        1. YARN-2637.40.patch
          84 kB
          Craig Welch
        2. YARN-2637.39.patch
          81 kB
          Craig Welch
        3. YARN-2637.38.patch
          81 kB
          Craig Welch
        4. YARN-2637.36.patch
          79 kB
          Craig Welch
        5. YARN-2637.32.patch
          77 kB
          Craig Welch
        6. YARN-2637.31.patch
          76 kB
          Craig Welch
        7. YARN-2637.30.patch
          52 kB
          Craig Welch
        8. YARN-2637.29.patch
          48 kB
          Craig Welch
        9. YARN-2637.28.patch
          46 kB
          Craig Welch
        10. YARN-2637.27.patch
          45 kB
          Craig Welch
        11. YARN-2637.26.patch
          23 kB
          Craig Welch
        12. YARN-2637.25.patch
          21 kB
          Craig Welch
        13. YARN-2637.23.patch
          35 kB
          Craig Welch
        14. YARN-2637.22.patch
          35 kB
          Craig Welch
        15. YARN-2637.21.patch
          19 kB
          Craig Welch
        16. YARN-2637.20.patch
          26 kB
          Craig Welch
        17. YARN-2637.19.patch
          33 kB
          Craig Welch
        18. YARN-2637.18.patch
          32 kB
          Craig Welch
        19. YARN-2637.17.patch
          31 kB
          Craig Welch
        20. YARN-2637.16.patch
          30 kB
          Craig Welch
        21. YARN-2637.15.patch
          28 kB
          Craig Welch
        22. YARN-2637.13.patch
          42 kB
          Craig Welch
        23. YARN-2637.12.patch
          42 kB
          Craig Welch
        24. YARN-2637.9.patch
          39 kB
          Craig Welch
        25. YARN-2637.7.patch
          36 kB
          Craig Welch
        26. YARN-2637.6.patch
          35 kB
          Craig Welch
        27. YARN-2637.2.patch
          11 kB
          Craig Welch
        28. YARN-2637.1.patch
          12 kB
          Craig Welch
        29. YARN-2637.0.patch
          11 kB
          Craig Welch

          Issue Links

            Activity

              People

              • Assignee:
                cwelch Craig Welch
                Reporter:
                leftnoteasy Wangda Tan
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: