[YARN-291] [Umbrella] Dynamic resource configuration

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: graceful, nodemanager, scheduler
• Labels: features
• Assignee: Junping Du
• Reporter: Junping Du
• Created: 23/Dec/12 08:17
• Updated: 25/Oct/19 20:26

Description

The current Hadoop YARN resource management logic assumes per node resource is static during the lifetime of the NM process. Allowing run-time configuration on per node resource will give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist with other workloads on the same hardware efficiently, whether or not the environment is virtualized. More background and design details can be found in attached proposal.

Issue Links

• blocks: YARN-914 (Umbrella) Support graceful decommission of nodemanager
• is related to: YARN-160 nodemanagers should obtain cpu/memory values from underlying OS
• relates to: YARN-45 [Preemption] Scheduler feedback to AM to release containers
• relates to: MAPREDUCE-4584 Umbrella: Preemption and restart of MapReduce tasks
• relates to: MAPREDUCE-4900 Dynamic configuration for task slots on TT
• relates to: YARN-3482 Report NM resource capacity in heartbeat
• supercedes: HADOOP-9165 Dynamic Resource/Slot Configuration on NM/TT

Sub-Tasks

• Dynamic node resource configuration: core scheduler changes: Closed
• Add updateNodeResource in ResourceManagerAdministrationProtocol: Closed
• Add Admin API for supporting node resource configuration in command line: Resolved
• REST API support for node resource configuration: Resolved
• JMX support for node resource configuration: Open
• Keep NM resource updated through dynamic resource config for RM/NM restart: Resolved
• In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.: Resolved
• Dynamic resource configuration feature can be configured to enable or disable and persistent on setting or not: Open
• Replace set resource change on RMNode/SchedulerNode directly with event notification.: Resolved
• Document Dynamic Resource Configuration feature: Open
• Add admin ACL check to AdminService#updateNodeResource(): Resolved
• ResouceOption's overcommitTimeout should be respected during resource update on NM: Resolved
• Dynamic NM Resources Configuration file should be simplified.: Open
• yarn rmadmin -updateNodeResource doesn't work: Resolved
• updateNodeResource CLI should not accept negative values for resource: Resolved
• RM Admin CLI for refreshNodesResources throws NPE when nothing is configured: Resolved
• NM side resource value should get updated if change applied in RM side: Resolved
• ContainersMonitorImpl need be adjusted when NM resource changed.: Resolved