[YARN-2492] (Clone of YARN-796) Allow for (admin) labels on nodes and resource-requests

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: api, client, resourcemanager
• Assignee: Unassigned
• Reporter: Wangda Tan
• Created: 03/Sep/14 08:47
• Updated: 25/Oct/19 20:25

Description

Since YARN-796 is a sub JIRA of YARN-397, this JIRA is used to create and track sub tasks and attach split patches for YARN-796.

Let's still keep over-all discussions on YARN-796.

Issue Links

• is cloned by: YARN-796 Allow for (admin) labels on nodes and resource-requests
• relates to: YARN-3894 RM startup should fail for wrong CS xml NodeLabel capacity configuration
• relates to: YARN-3971 Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery
• relates to: YARN-4454 NM to nodelabel mapping going wrong after RM restart
• relates to: YARN-4106 NodeLabels for NM in distributed mode is not updated even after clusterNodelabel addition in RM
• relates to: YARN-4176 Resync NM nodelabels with RM periodically for distributed nodelabels
• relates to: YARN-5788 Apps not activiated and AM limit resource in UI and REST not updated after -replaceLabelsOnNode
• relates to: MAPREDUCE-6304 Specifying node labels when submitting MR jobs
• relates to: YARN-3409 Support Node Attribute functionality
• relates to: YARN-4652 Skip label configuration and calculation of queue if nodelabel disabled
• relates to: YARN-4582 Label-related invalid resource request exception should be able to properly handled by application
• relates to: YARN-4465 SchedulerUtils#validateRequest for Label check should happen only when nodelabel enabled

Sub-Tasks

• API changes for users: Closed
• Node label manager API and storage implementations: Closed
• Allow admin specify labels from each NM (Distributed configuration): Resolved
• Changes for capacity scheduler to support allocate resource respect labels: Closed
• Fair scheduler should support strict node labels: Patch Available
• Respect labels in preemption policy of capacity scheduler for inter-queue preemption: Resolved
• Respect labels in preemption policy of fair scheduler: Open
• Miscellaneous changes in ResourceManager to support labels: Closed
• Changes in AMRMClient to support labels: Closed
• Changes in distributed shell to support specify labels: Closed
• Changes in RM Web UI to better show labels to end users: Closed
• Support get/add/remove/change labels in RM admin CLI: Closed
• Support get/add/remove/change labels in RM REST API: Closed
• Common server side PB changes (not include user API PB changes): Closed
• Add yarn queue CLI to get queue infos: Closed
• Resource on each label not correct when multiple NMs in a same host and some has label some not: Closed
• Ensure only single node labels specified in resource request / host, and node label expression only specified when resourceName=ANY: Closed
• Support continuously looking reserved container with node labels: Open
• Queue sorting in CapacityScheduler should consider node label: Resolved
• Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of AdminService: Closed
• Fix test timeout in TestResourceTrackerOnHA#testResourceTrackerOnHA: Closed
• Changes of RM node label manager default configuration: Closed
• rmadmin -replaceLabelsOnNode does not correctly parse port: Closed
• Support for disabling the Centralized NodeLabel validation in Distributed Node Label Configuration setup: Open
• Support script based NodeLabelsProvider Interface in Distributed Node Label Configuration Setup: Resolved
• CapacityScheduler should explicitly log when an accessible label has no capacity: Closed
• Fix NodeLabelsManager to properly handle node label modifications when distributed node label configuration enabled: Resolved
• Under some scenario, it is possible to end up with capacity scheduler configuration that uses labels that no longer exist: Closed
• Fix potential issues and code clean up for *NodeLabelsManager: Closed
• potential NPE in checkNodeLabelExpression of SchedulerUtils for nodeLabels.: Resolved
• use static variable (Resources.none()) for not-running Node.resource in CommonNodeLabelsManager to save memory.: Resolved
• RMAdminCLI node-labels-related args should be trimmed and checked before sending to RM: Closed
• YARN node CLI should display labels on returned node reports: Closed
• Create yarn cluster CLI to enable list node labels collection: Closed
• Resource Manager fails startup with HDFS label storage and secure cluster: Closed
• Remove MemoryNodeLabelsStore and add a way to enable/disable node labels feature: Closed
• Capacity of labels should be zero by default: Closed
• NodeLabels manager should trim all inputs for hosts and labels: Closed
• Add a test in TestRMRestart to make sure node labels will be recovered if it is enabled: Closed
• Add documentation for node labels feature: Closed
• CapacityScheduler should be notified when labels on nodes changed: Closed
• Support configuration based NodeLabelsProvider Service in Distributed Node Label Configuration Setup: Resolved
• Node to labels mapping should not transfer to lowercase when adding from RMAdminCLI: Closed
• Capacity Scheduler preemption policy should only consider capacity without labels temporarily: Closed
• Add a node-labels page in RM web UI: Closed
• NodeLabel operations in RMAdmin CLI get missing in help command.: Closed
• Provide API for client to retrieve label to node mapping: Closed
• Replaces labels on a host should update all NM's labels on that host: Closed
• (Refactoring) Merge internalAdd/Remove/ReplaceLabels to one method in CommonNodeLabelsManager: Resolved
• Better syntax for replaceLabelsOnNode in RMAdmin CLI: Closed
• Add API/Implementation to YarnClient to retrieve label-to-node mapping: Closed
• NodeLabelsManager implementation to retrieve label to node mapping: Closed
• Create common ResourceUsage class to track labeled resource usages in Capacity Scheduler: Closed
• Support load command from admin [Helps to load big set of labels]: Open
• Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues: Closed
• Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage to track used-resources-by-label.: Closed
• Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label: Closed
• RMNodeLabelsManager should remove node from node-to-label mapping when node becomes deactivated: Closed
• Respect labels in Capacity Scheduler when computing user-limit: Resolved
• Add non-exclusive node labels: Resolved
• Respect labels in CapacityScheduler when computing headroom: Resolved
• Max-AM-Resource-Percentage should respect node labels: Resolved
• Support RESTful API for getLabelsToNodes: Resolved
• Add non-exclusive node label API to RMAdmin protocol and NodeLabelsManager: Resolved
• Container should contains node-labels asked by original ResourceRequests: Resolved
• Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to track used-resources-by-label.: Resolved
• CapacityScheduler side changes to support non-exclusive node labels: Resolved
• Add node label usage in RM CapacityScheduler web UI: Closed
• Node label attributes (like exclusivity) should settable via addToClusterNodeLabels but shouldn't be changeable at runtime: Resolved
• Duplicate check for partitioned request on partitioned node: Open
• Error handling on NM reporting invalid NodeLabels in distributed Node Label configuration: Open
• Support intra-queue non-exclusive container allocation within a LeafQueue: Open
• Support return structured NodeLabel objects in REST API: Resolved
• NodeHeartbeatRequest/RegisterNodeManagerRequest should use NodeLabel object instead of String: Resolved
• Don't fail RM if queue's configured labels are not existed in cluster-node-labels: Closed
• CommonNodeLabelsManager should support NodeLabel instead of string label name when getting node-to-label/label-to-label mappings: Resolved
• Deprecate -directlyAccessNodeLabelStore in RMAdminCLI: Resolved
• Add label-type and Improve "DEFAULT_PARTITION" in Node Labels Page: Resolved
• Move load labels from storage from serviceInit to serviceStart to make it works with RM HA case.: Closed
• Support of NodeLabel object instead of plain String in YarnClient side.: Resolved
• RMWebServices api's should use updated api from CommonNodeLabelsManager to get NodeLabel object: Resolved
• CapacityScheduler should trim default_node_label_expression: Closed
• We should be able to remove node-label if there's no queue can use it.: Open
• Expose app/am/queue's node-label-expression to RM web UI / CLI / REST-API: Resolved
• Node-label-expression should be included by ResourceRequestPBImpl.toString: Resolved
• FileSystemNodeLabelsStore should make sure edit log file closed when exception is thrown: Resolved
• Application moveToQueue should check NodeLabel permission: Resolved
• Support NodeLabelsProvider at Resource Manager side: Resolved
• Container shouldn't be killed when node's label updated.: Resolved
• Add Documentation for Distributed and Delegated-Centralized Node Labels feature: Resolved
• RM container allocation delayed incase of app submitted to Nodelabel partition: Resolved
• CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API: Resolved
• RMNodeLabels Manager Need to verify and replace node labels for the only modified Node Label Mappings in the request: Resolved
• Support max-am-resource-percentage per label partition for User: Resolved
• Node Label should support pluggable storage: Resolved
• Add documentation for delegated-centralized node labels feature: Resolved
• AM max resource configuration per partition to be displayed/updated correctly in UI and in various partition related metrics: Resolved
• Support node label store in non-appendable file system: Resolved
• Scheduler Web Ui shows max capacity for the queue is 100% but when we submit application doesnt get assigned: Open
• Pluggable sharing policy for Partition Node Label resources: Open
• Available Resource calculation for a queue is not correct when used with labels: Resolved
• Application level node labels stats to be available in REST: Resolved
• Fix improper Queues sorting in PartitionedQueueComparator when accessible-node-labels=*: Resolved
• Need to support Reservation for partitions other than DEFAULT_LABEL: Resolved
• Scheduler UI/Metrics need to consider cases like non-queue label mappings: Resolved
• Add documentation for the Node Label features supported in 2.6: Open
• Support node labels store in zookeeper: Resolved
• Improve non-exclusive node partition resource allocation in Capacity Scheduler: Resolved
• Fair Scheduler: respect node labels at resource request level: Open
• Add configuration to initialize RM with configured labels.: Patch Available