Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2492

(Clone of YARN-796) Allow for (admin) labels on nodes and resource-requests

    Details

    • Type: Task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: api, client, resourcemanager
    • Labels:
      None

      Description

      Since YARN-796 is a sub JIRA of YARN-397, this JIRA is used to create and track sub tasks and attach split patches for YARN-796.

      Let's still keep over-all discussions on YARN-796.

        Attachments

          Issue Links

          1.
          API changes for users Sub-task Closed Wangda Tan
          2.
          Node label manager API and storage implementations Sub-task Closed Wangda Tan
          3.
          Allow admin specify labels from each NM (Distributed configuration) Sub-task Resolved Naganarasimha G R
          4.
          Changes for capacity scheduler to support allocate resource respect labels Sub-task Closed Wangda Tan
          5.
          Fair scheduler should support strict node labels Sub-task Patch Available Daniel Templeton
          6.
          Respect labels in preemption policy of capacity scheduler for inter-queue preemption Sub-task Resolved Wangda Tan
          7.
          Respect labels in preemption policy of fair scheduler Sub-task Open Zhaohui Xin
          8.
          Miscellaneous changes in ResourceManager to support labels Sub-task Closed Wangda Tan
          9.
          Changes in AMRMClient to support labels Sub-task Closed Wangda Tan
          10.
          Changes in distributed shell to support specify labels Sub-task Closed Wangda Tan
          11.
          Changes in RM Web UI to better show labels to end users Sub-task Closed Wangda Tan
          12.
          Support get/add/remove/change labels in RM admin CLI Sub-task Closed Wangda Tan
          13.
          Support get/add/remove/change labels in RM REST API Sub-task Closed Craig Welch
          14.
          Common server side PB changes (not include user API PB changes) Sub-task Closed Wangda Tan
          15.
          Add yarn queue CLI to get queue infos Sub-task Closed Sunil Govindan
          16.
          Resource on each label not correct when multiple NMs in a same host and some has label some not Sub-task Closed Wangda Tan
          17.
          Ensure only single node labels specified in resource request / host, and node label expression only specified when resourceName=ANY Sub-task Closed Wangda Tan
          18.
          Support continuously looking reserved container with node labels Sub-task Open Unassigned
          19.
          Queue sorting in CapacityScheduler should consider node label Sub-task Resolved Wangda Tan
          20.
          Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of AdminService Sub-task Closed Wangda Tan
          21.
          Fix test timeout in TestResourceTrackerOnHA#testResourceTrackerOnHA Sub-task Closed Wangda Tan
          22.
          Changes of RM node label manager default configuration Sub-task Closed Wangda Tan
          23.
          rmadmin -replaceLabelsOnNode does not correctly parse port Sub-task Closed Naganarasimha G R
          24.
          Support for disabling the Centralized NodeLabel validation in Distributed Node Label Configuration setup Sub-task Open Naganarasimha G R
          25.
          Support script based NodeLabelsProvider Interface in Distributed Node Label Configuration Setup Sub-task Resolved Naganarasimha G R
          26.
          CapacityScheduler should explicitly log when an accessible label has no capacity Sub-task Closed Wangda Tan
          27.
          Fix NodeLabelsManager to properly handle node label modifications when distributed node label configuration enabled Sub-task Resolved Naganarasimha G R
          28.
          Under some scenario, it is possible to end up with capacity scheduler configuration that uses labels that no longer exist Sub-task Closed Wangda Tan
          29.
          Fix potential issues and code clean up for *NodeLabelsManager Sub-task Closed zhihai xu
          30.
          potential NPE in checkNodeLabelExpression of SchedulerUtils for nodeLabels. Sub-task Resolved zhihai xu
          31.
          use static variable (Resources.none()) for not-running Node.resource in CommonNodeLabelsManager to save memory. Sub-task Resolved zhihai xu
          32.
          RMAdminCLI node-labels-related args should be trimmed and checked before sending to RM Sub-task Closed Rohith Sharma K S
          33.
          YARN node CLI should display labels on returned node reports Sub-task Closed Wangda Tan
          34.
          Create yarn cluster CLI to enable list node labels collection Sub-task Closed Wangda Tan
          35.
          Resource Manager fails startup with HDFS label storage and secure cluster Sub-task Closed Wangda Tan
          36.
          Remove MemoryNodeLabelsStore and add a way to enable/disable node labels feature Sub-task Closed Wangda Tan
          37.
          Capacity of labels should be zero by default Sub-task Closed Wangda Tan
          38.
          NodeLabels manager should trim all inputs for hosts and labels Sub-task Closed Wangda Tan
          39.
          Add a test in TestRMRestart to make sure node labels will be recovered if it is enabled Sub-task Closed Rohith Sharma K S
          40.
          Add documentation for node labels feature Sub-task Closed Wangda Tan
          41.
          CapacityScheduler should be notified when labels on nodes changed Sub-task Closed Wangda Tan
          42.
          Support configuration based NodeLabelsProvider Service in Distributed Node Label Configuration Setup Sub-task Resolved Naganarasimha G R
          43.
          Node to labels mapping should not transfer to lowercase when adding from RMAdminCLI Sub-task Closed Wangda Tan
          44.
          Capacity Scheduler preemption policy should only consider capacity without labels temporarily Sub-task Closed Mayank Bansal
          45.
          Add a node-labels page in RM web UI Sub-task Closed Wangda Tan
          46.
          NodeLabel operations in RMAdmin CLI get missing in help command. Sub-task Closed Varun Saxena
          47.
          Provide API for client to retrieve label to node mapping Sub-task Closed Varun Saxena
          48.
          Replaces labels on a host should update all NM's labels on that host Sub-task Closed Wangda Tan
          49.
          (Refactoring) Merge internalAdd/Remove/ReplaceLabels to one method in CommonNodeLabelsManager Sub-task Resolved Wangda Tan
          50.
          Better syntax for replaceLabelsOnNode in RMAdmin CLI Sub-task Closed Rohith Sharma K S
          51.
          Add API/Implementation to YarnClient to retrieve label-to-node mapping Sub-task Closed Varun Saxena
          52.
          NodeLabelsManager implementation to retrieve label to node mapping Sub-task Closed Varun Saxena
          53.
          Create common ResourceUsage class to track labeled resource usages in Capacity Scheduler Sub-task Closed Wangda Tan
          54.
          Support load command from admin [Helps to load big set of labels] Sub-task Open Sunil Govindan
          55.
          Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues Sub-task Closed Wangda Tan
          56.
          Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage to track used-resources-by-label. Sub-task Closed Wangda Tan
          57.
          Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label Sub-task Closed Wangda Tan
          58.
          RMNodeLabelsManager should remove node from node-to-label mapping when node becomes deactivated Sub-task Closed Wangda Tan
          59.
          Respect labels in Capacity Scheduler when computing user-limit Sub-task Resolved Wangda Tan
          60.
          Add non-exclusive node labels Sub-task Resolved Wangda Tan
          61.
          Respect labels in CapacityScheduler when computing headroom Sub-task Resolved Naganarasimha G R
          62.
          Max-AM-Resource-Percentage should respect node labels Sub-task Resolved Sunil Govindan
          63.
          Support RESTful API for getLabelsToNodes Sub-task Resolved Naganarasimha G R
          64.
          Add non-exclusive node label API to RMAdmin protocol and NodeLabelsManager Sub-task Resolved Wangda Tan
          65.
          Container should contains node-labels asked by original ResourceRequests Sub-task Resolved Wangda Tan
          66.
          Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to track used-resources-by-label. Sub-task Resolved Wangda Tan
          67.
          CapacityScheduler side changes to support non-exclusive node labels Sub-task Resolved Wangda Tan
          68.
          Add node label usage in RM CapacityScheduler web UI Sub-task Closed Naganarasimha G R
          69.
          Node label attributes (like exclusivity) should settable via addToClusterNodeLabels but shouldn't be changeable at runtime Sub-task Resolved Wangda Tan
          70.
          Duplicate check for partitioned request on partitioned node Sub-task Open Jian He
          71.
          Error handling on NM reporting invalid NodeLabels in distributed Node Label configuration Sub-task Open Naganarasimha G R
          72.
          Support intra-queue non-exclusive container allocation within a LeafQueue Sub-task Open Wangda Tan
          73.
          Support return structured NodeLabel objects in REST API Sub-task Resolved Sunil Govindan
          74.
          NodeHeartbeatRequest/RegisterNodeManagerRequest should use NodeLabel object instead of String Sub-task Resolved Naganarasimha G R
          75.
          Don't fail RM if queue's configured labels are not existed in cluster-node-labels Sub-task Closed Wangda Tan
          76.
          CommonNodeLabelsManager should support NodeLabel instead of string label name when getting node-to-label/label-to-label mappings Sub-task Resolved Sunil Govindan
          77.
          Deprecate -directlyAccessNodeLabelStore in RMAdminCLI Sub-task Resolved Naganarasimha G R
          78.
          Add label-type and Improve "DEFAULT_PARTITION" in Node Labels Page Sub-task Resolved Naganarasimha G R
          79.
          Move load labels from storage from serviceInit to serviceStart to make it works with RM HA case. Sub-task Closed Wangda Tan
          80.
          Support of NodeLabel object instead of plain String in YarnClient side. Sub-task Resolved Sunil Govindan
          81.
          RMWebServices api's should use updated api from CommonNodeLabelsManager to get NodeLabel object Sub-task Resolved Sunil Govindan
          82.
          CapacityScheduler should trim default_node_label_expression Sub-task Closed Sunil Govindan
          83.
          We should be able to remove node-label if there's no queue can use it. Sub-task Open Wangda Tan
          84.
          Expose app/am/queue's node-label-expression to RM web UI / CLI / REST-API Sub-task Resolved Naganarasimha G R
          85.
          Node-label-expression should be included by ResourceRequestPBImpl.toString Sub-task Resolved Xianyin Xin
          86.
          FileSystemNodeLabelsStore should make sure edit log file closed when exception is thrown Sub-task Resolved Dian Fu
          87.
          Application moveToQueue should check NodeLabel permission Sub-task Resolved Bibin A Chundatt
          88.
          Support NodeLabelsProvider at Resource Manager side Sub-task Resolved Dian Fu
          89.
          Container shouldn't be killed when node's label updated. Sub-task Resolved Wangda Tan
          90.
          Add Documentation for Distributed and Delegated-Centralized Node Labels feature Sub-task Resolved Naganarasimha G R
          91.
          RM container allocation delayed incase of app submitted to Nodelabel partition Sub-task Resolved Bibin A Chundatt
          92.
          CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API Sub-task Resolved Naganarasimha G R
          93.
          RMNodeLabels Manager Need to verify and replace node labels for the only modified Node Label Mappings in the request Sub-task Resolved Naganarasimha G R
          94.
          Support max-am-resource-percentage per label partition for User Sub-task Resolved Sunil Govindan
          95.
          Node Label should support pluggable storage Sub-task Resolved Wangda Tan
          96.
          Add documentation for delegated-centralized node labels feature Sub-task Resolved Dian Fu
          97.
          AM max resource configuration per partition to be displayed/updated correctly in UI and in various partition related metrics Sub-task Resolved Sunil Govindan
          98.
          Support node label store in non-appendable file system Sub-task Resolved Wangda Tan
          99.
          Scheduler Web Ui shows max capacity for the queue is 100% but when we submit application doesnt get assigned Sub-task Open Naganarasimha G R
          100.
          Pluggable sharing policy for Partition Node Label resources Sub-task Open Naganarasimha G R
          101.
          Available Resource calculation for a queue is not correct when used with labels Sub-task Resolved Sunil Govindan
          102.
          Application level node labels stats to be available in REST Sub-task Resolved Bibin A Chundatt
          103.
          Fix improper Queues sorting in PartitionedQueueComparator when accessible-node-labels=* Sub-task Resolved Naganarasimha G R
          104.
          Need to support Reservation for partitions other than DEFAULT_LABEL Sub-task Resolved Unassigned
          105.
          Scheduler UI/Metrics need to consider cases like non-queue label mappings Sub-task Resolved Sunil Govindan
          106.
          Add documentation for the Node Label features supported in 2.6 Sub-task Open Naganarasimha G R
          107.
          Support node labels store in zookeeper Sub-task Resolved jialei weng
          108.
          Improve non-exclusive node partition resource allocation in Capacity Scheduler Sub-task Resolved Sunil Govindan
          109.
          Fair Scheduler: respect node labels at resource request level Sub-task Open Ashwin Shankar
          110.
          Add configuration to initialize RM with configured labels. Sub-task Patch Available Abhishek Modi

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                leftnoteasy Wangda Tan
              • Votes:
                0 Vote for this issue
                Watchers:
                47 Start watching this issue

                Dates

                • Created:
                  Updated: