Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7069

[Umbrella] YARN Resource Profiles phase#2 enhancements

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • nodemanager, resourcemanager
    • None

    Description

      This ticket to is to track further enhancements to resource profile work happening in YARN-3926

      Attachments

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. YARN-7314 Multiple Resource manager test cases failing

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is related to

          Task - A task that needs to be done. YARN-8200 Backport resource types/GPU features to branch-3.0/branch-2

          • Major - Major loss of function.
          • Resolved
          relates to

          New Feature - A new feature of the product, which has yet to be developed. YARN-3926 [Umbrella] Extend the YARN resource model for easier resource-type management and profiles

          • Major - Major loss of function.
          • Resolved
          1.
          		 Add support for increase and decrease of container resources with resource profiles Sub-task Patch Available Manikandan R
          2.
          		 Update CapacitySchedulerConfiguration minimum and maximum calculations to consider all resource types Sub-task Patch Available Unassigned
          3.
          		 Update web UIs to reflect multiple resource types Sub-task Open Varun Vasudev
          4.
          		 Add support for dynamic resource updates with multiple resource types Sub-task Open Manikandan R
          5.
          		 Add support for resource profiles in MapReduce Sub-task Patch Available Varun Vasudev
          6.
          		 Update fair scheduler policies to be aware of resource types Sub-task Resolved Daniel Templeton
          7.
          		 ResourceTypes should be renamed Sub-task Open Zhaohui Xin
          8.
          		 Node information page in the old web UI should report resource types Sub-task Resolved Gergely Novák
          9.
          		 Use LightWeightedResource when number of resource types more than two Sub-task Resolved Sunil G
          10.
          		 Cluster Metrics API should report resource types information Sub-task Open Unassigned
          11.
          		 Add support for individual resource types requests in MapReduce Sub-task Resolved Gergo Repas
          12.
          		 Cluster Node API and Cluster Nodes API should report resource types information Sub-task Patch Available Manikandan R
          13.
          		 Clean up ResourceUtils.setMinimumAllocationForMandatoryResources() and setMaximumAllocationForMandatoryResources() Sub-task Resolved Manikandan R
          14.
          		 Clean up different ways to create resources Sub-task Resolved Riya Khandelwal
          15.
          		 DominantResourceCalculator.isAnyMajorResourceZero() should test all resources Sub-task Resolved Sunil G
          16.
          		 Improve use of ResourceNotFoundException in resource types code Sub-task Resolved Szilard Nemeth
          17.
          		 Recover all resources when NM restart Sub-task Resolved Yang Wang
          18.
          		 TestResourceUtils to be moved to hadoop-yarn-api package Sub-task Open Zhaohui Xin
          19.
          		 FileNotFound handling in ResourceUtils is inconsistent Sub-task Resolved Daniel Templeton
          20.
          		 Normalize unit of resource objects in RM to avoid unit conversion in critical path Sub-task Resolved Manikandan R
          21.
          		 Container REST endpoints should report resource types Sub-task Resolved Daniel Templeton
          22.
          		 Remove class ResourceType Sub-task Resolved Sunil G
          23.
          		 Support multiple resource types in rmadmin updateNodeResource command Sub-task Resolved Manikandan R
          24.
          		 Log improvements for the ResourceUtils Sub-task Resolved Sunil G
          25.
          		 IllegalArgumentException in AMRMClient Sub-task Resolved Unassigned
          26.
          		 Cleanup usages of ResourceProfiles Sub-task Resolved Wangda Tan
          27.
          		 Support to specify values of different resource types in DistributedShell for easier testing Sub-task Resolved Gergely Novák
          28.
          		 Retrospect Resource Profile Behavior for overriding capability Sub-task Resolved Wangda Tan
          29.
          		 convertToProtoFormat(Resource r) is not setting for all resource types Sub-task Resolved lovekesh bansal
          30.
          		 Allow client/AM update supported resource types via YARN APIs Sub-task Resolved Sunil G
          31.
          		 [Resource Profiles] getMaximumAllocationPerQueue to account for all resources Sub-task Patch Available lovekesh bansal
          32.
          		 Cleaning up the usage of Resources and ResourceCalculator Sub-task Patch Available lovekesh bansal
          33.
          		 ResourceUtils allows yarn.nodemanager.resource-types.memory-mb and .vcores to override yarn.nodemanager.resource.memory-mb and .cpu-vcores Sub-task Resolved lovekesh bansal
          34.
          		 Improve the resource types docs Sub-task Resolved Daniel Templeton
          35.
          		 Improve performance of DRF comparisons for resource types in fair scheduler Sub-task Resolved Daniel Templeton
          36.
          		 Node resource is not parsed correctly for resource names containing dot Sub-task Resolved Gergely Novák
          37.
          		 [YARN-7069] Limit format of resource type name Sub-task Resolved Wangda Tan
          38.
          		 Node manager should allow resource units to be lower cased Sub-task Open Zhaohui Xin
          39.
          		 Node updates don't update the maximum cluster capability for resources other than CPU and memory Sub-task Resolved Daniel Templeton
          40.
          		 Fair scheduler configuration should allow resource types in the minResources and maxResources properties Sub-task Resolved Szilard Nemeth
          41.
          		 It should be possible to specify resource types in the fair scheduler increment value Sub-task Resolved Gergo Repas
          42.
          		 Resource configs not parsed correctly in MapReduce for resources with dot Sub-task Patch Available Jonathan Hung
          43.
          		 RM gives YARNFeatureNotEnabledException even when resource profile feature is not enabled Sub-task Patch Available Manikandan R
          44.
          		 Should fail RM if 3rd resource type is configured but RM uses DefaultResourceCalculator Sub-task Resolved Zian Chen
          45.
          		 Add Resource reference to RM's NodeInfo object so REST API can get non memory/vcore resource usages. Sub-task Resolved Sunil G
          46.
          		 Fix UT failure TestRMWebServiceAppsNodelabel#testAppsRunning Sub-task Resolved Sunil G
          47.
          		 Adding profile capability in resourceReq in LocalityMulticastAMRMProxyPolicy Sub-task Patch Available lovekesh bansal
          48.
          		 Fair Scheduler stops scheduling when a queue is configured only CPU and memory Sub-task Patch Available Sen Zhao
          49.
          		 Add resource-types.xml to configuration when refreshing maximum allocation Sub-task Resolved Jonathan Hung

          Activity

            People

              Unassigned Unassigned
              sunilg Sunil G
              Votes:
              0 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

                Created:
                Updated: