[YARN-3926] [Umbrella] Extend the YARN resource model for easier resource-type management and profiles

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1.0
• Component/s: nodemanager, resourcemanager
• Hadoop Flags: Reviewed
• Assignee: Varun Vasudev
• Reporter: Varun Vasudev
• Created: 15/Jul/15 03:12
• Updated: 27/Jun/18 01:23
• Resolved: 12/Sep/17 19:50

Description

Currently, there are efforts to add support for various resource-types such as disk(YARN-2139), network(YARN-2140), and HDFS bandwidth(YARN-2681). These efforts all aim to add support for a new resource type and are fairly involved efforts. In addition, once support is added, it becomes harder for users to specify the resources they need. All existing jobs have to be modified, or have to use the minimum allocation.

This ticket is a proposal to extend the YARN resource model to a more flexible model which makes it easier to support additional resource-types. It also considers the related aspect of “resource profiles” which allow users to easily specify the various resources they need for any given container.

This feature is already merged to trunk, please go to YARN-7069 for all pending items.

Issue Links

• is depended upon by: YARN-6690 Consolidate NM overallocation thresholds with ResourceTypes
• is depended upon by: SPARK-20327 Add CLI support for YARN custom resources, like GPUs
• is related to: YARN-2139 [Umbrella] Support for Disk as a Resource in YARN
• is related to: YARN-2140 Add support for network IO isolation/scheduling for containers
• is related to: YARN-2681 Support bandwidth enforcement for containers while reading from HDFS
• is related to: YARN-7069 [Umbrella] YARN Resource Profiles phase#2 enhancements
• is required by: YARN-5983 [Umbrella] Support for FPGA as a Resource in YARN
• is required by: YARN-6223 [Umbrella] Natively support GPU configuration/discovery/scheduling/isolation on YARN
• relates to: YARN-6720 Support updating FPGA related constraint node label after FPGA device re-configuration
• relates to: YARN-6589 Recover all resources when NM restart
• relates to: YARN-4122 Add support for GPU as a resource
• relates to: YARN-8200 Backport resource types/GPU features to branch-3.0/branch-2

Sub-Tasks

• Add support for multiple resource types in the Resource class: Resolved
• Extend DominantResourceCalculator to account for all resources: Resolved
• Add support to read resource types from a config file: Resolved
• Add support for binary units: Resolved
• Add support for resource types in the nodemanager: Resolved
• Update DominantResourceCalculator to consider all resource types in calculations: Resolved
• Update the Resources class to consider all resource types: Resolved
• Add support for resource profiles: Resolved
• Add support for resource profiles in distributed shell: Resolved
• Add manager class for resource profiles: Resolved
• Implement APIs to get resource profiles from the RM: Resolved
• Update resource usage and preempted resource calculations to take into account all resource types: Resolved
• [YARN-3926] Performance improvements in resource profile branch with respect to SLS: Resolved
• DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources: Resolved
• Fix build for YARN-3926 branch: Resolved
• ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter: Resolved
• ResourceUtils.getResourceTypes() reloads the resource configuration every time: Resolved
• DominantResourceCalculator.resourceNames should be a constant: Resolved
• Resource.compareTo() is implemented twice: Resolved
• Resource.SimpleResource does not implement the new Resource methods: Resolved
• ResourcePBImpl imports cleanup: Resolved
• Improve performance of resource profile branch: Resolved
• Add Client API to get all supported resource types from RM: Resolved
• Improve API implementation in Resources and DominantResourceCalculator class: Resolved
• ResourceProfilesManagerImpl is missing @Overrides on methods: Resolved
• ResourceUtils.DISALLOWED_NAMES check is duplicated: Resolved
• ResourceUtils.checkMandatoryResources() should also ensure that no min or max is set for vcores or memory: Resolved
• ResourceProfilesManagerImpl.parseResource() has no need of the key parameter: Resolved
• Remove last uses of Long from resource types code: Resolved
• merge related work for YARN-3926 branch: Patch Available
• Performance optimizations in Resource and ResourceUtils class: Resolved
• Fix javac and javadoc errors in YARN-3926 branch: Resolved
• Clean up unit tests after YARN-6610: Resolved
• Cleanup ResourceProfileManager: Resolved
• Document Resource Profiles feature: Resolved
• Optimize ResourceType information display in UI: Resolved
• Improve log message in ResourceUtils: Resolved
• Additional Performance Improvement for Resource Profile Feature: Resolved
• Move newly added APIs to unstable in YARN-3926 branch: Resolved
• Address performance issue when 3+ resource types configured in the system: Resolved