Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3926

[Umbrella] Extend the YARN resource model for easier resource-type management and profiles

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.0
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, there are efforts to add support for various resource-types such as disk(YARN-2139), network(YARN-2140), and HDFS bandwidth(YARN-2681). These efforts all aim to add support for a new resource type and are fairly involved efforts. In addition, once support is added, it becomes harder for users to specify the resources they need. All existing jobs have to be modified, or have to use the minimum allocation.

      This ticket is a proposal to extend the YARN resource model to a more flexible model which makes it easier to support additional resource-types. It also considers the related aspect of “resource profiles” which allow users to easily specify the various resources they need for any given container.

      This feature is already merged to trunk, please go to YARN-7069 for all pending items.

        Attachments

          Issue Links

          1.
          Add support for multiple resource types in the Resource class Sub-task Resolved Varun Vasudev
          2.
          Extend DominantResourceCalculator to account for all resources Sub-task Resolved Varun Vasudev
          3.
          Add support to read resource types from a config file Sub-task Resolved Varun Vasudev
          4.
          Add support for binary units Sub-task Resolved Varun Vasudev
          5.
          Add support for resource types in the nodemanager Sub-task Resolved Varun Vasudev
          6.
          Update DominantResourceCalculator to consider all resource types in calculations Sub-task Resolved Varun Vasudev
          7.
          Update the Resources class to consider all resource types Sub-task Resolved Varun Vasudev
          8.
          Add support for resource profiles Sub-task Resolved Varun Vasudev
          9.
          Add support for resource profiles in distributed shell Sub-task Resolved Varun Vasudev
          10.
          Add manager class for resource profiles Sub-task Resolved Varun Vasudev
          11.
          Implement APIs to get resource profiles from the RM Sub-task Resolved Varun Vasudev
          12.
          Update resource usage and preempted resource calculations to take into account all resource types Sub-task Resolved Varun Vasudev
          13.
          [YARN-3926] Performance improvements in resource profile branch with respect to SLS Sub-task Resolved Varun Vasudev
          14.
          DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources Sub-task Resolved Daniel Templeton
          15.
          Fix build for YARN-3926 branch Sub-task Resolved Varun Vasudev
          16.
          ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter Sub-task Resolved Yu-Tang Lin
          17.
          ResourceUtils.getResourceTypes() reloads the resource configuration every time Sub-task Resolved Unassigned
          18.
          DominantResourceCalculator.resourceNames should be a constant Sub-task Resolved Unassigned
          19.
          Resource.compareTo() is implemented twice Sub-task Resolved Daniel Templeton
          20.
          Resource.SimpleResource does not implement the new Resource methods Sub-task Resolved Unassigned
          21.
          ResourcePBImpl imports cleanup Sub-task Resolved Yeliang Cang
          22.
          Improve performance of resource profile branch Sub-task Resolved Sunil Govindan
          23.
          Add Client API to get all supported resource types from RM Sub-task Resolved Sunil Govindan
          24.
          Improve API implementation in Resources and DominantResourceCalculator class Sub-task Resolved Sunil Govindan
          25.
          ResourceProfilesManagerImpl is missing @Overrides on methods Sub-task Resolved Sunil Govindan
          26.
          ResourceUtils.DISALLOWED_NAMES check is duplicated Sub-task Resolved Manikandan R
          27.
          ResourceUtils.checkMandatoryResources() should also ensure that no min or max is set for vcores or memory Sub-task Resolved Unassigned
          28.
          ResourceProfilesManagerImpl.parseResource() has no need of the key parameter Sub-task Resolved Manikandan R
          29.
          Remove last uses of Long from resource types code Sub-task Resolved Daniel Templeton
          30.
          merge related work for YARN-3926 branch Sub-task Patch Available Daniel Templeton
          31.
          Performance optimizations in Resource and ResourceUtils class Sub-task Resolved Wangda Tan
          32.
          Fix javac and javadoc errors in YARN-3926 branch Sub-task Resolved Sunil Govindan
          33.
          Clean up unit tests after YARN-6610 Sub-task Resolved Daniel Templeton
          34.
          Cleanup ResourceProfileManager Sub-task Resolved Wangda Tan
          35.
          Document Resource Profiles feature Sub-task Resolved Sunil Govindan
          36.
          Optimize ResourceType information display in UI Sub-task Resolved Wangda Tan
          37.
          Improve log message in ResourceUtils Sub-task Resolved Sunil Govindan
          38.
          Additional Performance Improvement for Resource Profile Feature Sub-task Resolved Wangda Tan
          39.
          Move newly added APIs to unstable in YARN-3926 branch Sub-task Resolved Wangda Tan
          40.
          Address performance issue when 3+ resource types configured in the system Sub-task Resolved Wangda Tan

            Activity

              People

              • Assignee:
                vvasudev Varun Vasudev
                Reporter:
                vvasudev Varun Vasudev
              • Votes:
                1 Vote for this issue
                Watchers:
                74 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: