Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8468

Enable the use of queue based maximum container allocation limit and implement it in FairScheduler

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.1.0
    • 3.2.0
    • fairscheduler, scheduler
    • None
    • Reviewed

    Description

      When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb" to limit the overall size of a container. This applies globally to all containers and cannot be limited by queue or and is not scheduler dependent.

      The goal of this ticket is to allow this value to be set on a per queue basis.

      The use case: User has two pools, one for ad hoc jobs and one for enterprise apps. User wants to limit ad hoc jobs to small containers but allow enterprise apps to request as many resources as needed. Setting yarn.scheduler.maximum-allocation-mb sets a default value for maximum container size for all queues and setting maximum resources per queue with “maxContainerResources” queue config value.
      NOTE: "maxContainerResources" has been changed to "maxContainerAllocation" as per the review comments.

      Suggested solution:

      All the infrastructure is already in the code. We need to do the following:

      • add the setting to the queue properties for all queue types (parent and leaf), this will cover dynamically created queues.
      • if we set it on the root we override the scheduler setting and we should not allow that.
      • make sure that queue resource cap can not be larger than scheduler max resource cap in the config.
      • implement getMaximumResourceCapability(String queueName) in the FairScheduler
      • implement getMaximumResourceCapability(String queueName) in both FSParentQueue and FSLeafQueue as follows
      • expose the setting in the queue information in the RM web UI.
      • expose the setting in the metrics etc for the queue.
      • Enforce the use of queue based maximum allocation limit if it is available, if not use the general scheduler level setting
        • Use it during validation and normalization of requests in scheduler.allocate, app submit and resource request

      Attachments

        1. YARN-8468-branch-3.1.022.patch
          135 kB
          Weiwei Yang
        2. YARN-8468-branch-3.1.021.patch
          135 kB
          Weiwei Yang
        3. YARN-8468-branch-3.1.020.patch
          1.0 kB
          Weiwei Yang
        4. YARN-8468-branch-3.1.019.patch
          135 kB
          Antal Bálint Steinbach
        5. YARN-8468-branch-3.1.018.patch
          135 kB
          Antal Bálint Steinbach
        6. YARN-8468.018.patch
          134 kB
          Antal Bálint Steinbach
        7. YARN-8468.017.patch
          134 kB
          Antal Bálint Steinbach
        8. YARN-8468.016.patch
          133 kB
          Antal Bálint Steinbach
        9. YARN-8468.015.patch
          133 kB
          Antal Bálint Steinbach
        10. YARN-8468.014.patch
          140 kB
          Antal Bálint Steinbach
        11. YARN-8468.013.patch
          165 kB
          Antal Bálint Steinbach
        12. YARN-8468.012.patch
          156 kB
          Antal Bálint Steinbach
        13. YARN-8468.011.patch
          156 kB
          Antal Bálint Steinbach
        14. YARN-8468.010.patch
          156 kB
          Antal Bálint Steinbach
        15. YARN-8468.009.patch
          157 kB
          Antal Bálint Steinbach
        16. YARN-8468.008.patch
          151 kB
          Antal Bálint Steinbach
        17. YARN-8468.007.patch
          146 kB
          Antal Bálint Steinbach
        18. YARN-8468.006.patch
          90 kB
          Antal Bálint Steinbach
        19. YARN-8468.005.patch
          88 kB
          Antal Bálint Steinbach
        20. YARN-8468.004.patch
          75 kB
          Antal Bálint Steinbach
        21. YARN-8468.003.patch
          58 kB
          Antal Bálint Steinbach
        22. YARN-8468.002.patch
          58 kB
          Antal Bálint Steinbach
        23. YARN-8468.001.patch
          59 kB
          Antal Bálint Steinbach
        24. YARN-8468.000.patch
          56 kB
          Antal Bálint Steinbach

        Issue Links

          Activity

            People

              bsteinbach Antal Bálint Steinbach
              bsteinbach Antal Bálint Steinbach
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: