[YARN-10370] [Umbrella] Reduce the feature gap between FS Placement Rules and CS Queue Mapping rules

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: yarn
• Labels: capacity-scheduler capacityscheduler
• Assignee: Gergely Pollák
• Reporter: Gergely Pollák
• Created: 28/Jul/20 13:53
• Updated: 08/Apr/21 16:23
• Resolved: 08/Apr/21 16:20

Description

To continue closing the feature gaps between Fair Scheduler and Capacity Scheduler to help users migrate between the scheduler more easy, we need to add some of the Fair Scheduler placement rules to the capacity scheduler's queue mapping functionality.

With Szilard Nemeth and Peter Bacsko we've created the following design docs about the proposed changes.

Sub-Tasks

• Create variable context class for CS queue mapping rules: Resolved
• Create MappingRule class to represent each CS mapping rule: Resolved
• Create Matchers for CS mapping rules: Resolved
• Create Actions for CS mapping rules: Resolved
• CS Mapping rule config parser should return MappingRule objects: Resolved
• Create a class that covers the functionality of UserGroupMappingPlacementRule and AppNameMappingPlacementRule using the new mapping rules: Resolved
• Create new JSON schema for Placement Rules: Resolved
• Implement logic which returns MappingRule objects based on mapping rules: Resolved
• Extract MockQueueHierarchyBuilder to a separate class: Resolved
• Improve MockQueueHierarchyBuilder to detect queue ambiguity: Resolved
• Create an allowCreate flag for MappingRuleAction: Resolved
• Move CS placement rule related changes to a separate package: Resolved
• Change fs2cs to generate mapping rules in the new format: Resolved
• Create a group matcher which checks ALL groups of the user: Resolved
• Update CS MappingRule documentation with the new format and features: Resolved
• Adapt existing AppName and UserGroupMapping unittests to ensure backwards compatibility: Resolved
• Replace the legacy placement engine in CS with the new one: Resolved
• Add applicationName policy: Resolved
• Add a configuration switch to change between legacy and JSON placement rule format: Resolved
• Add the capability to fs2cs to write the converted placement rules inside capacity-scheduler.xml: Resolved
• RMAppManager CS Placement ignores parent path: Resolved
• Fix dependency exclusion problem in poms: Resolved
• Create a class which can convert from legacy mapping rule format to the new JSON format: Resolved
• CSMappingPlacementRule should not create new instance of Groups: Resolved
• Fix findbugs issue introduced in YARN-10585: Resolved
• CS Mapping Rule %specified rule catches default submissions: Resolved
• CSMapping rule can return paths with empty parts: Resolved
• Improve CS MappingRule %secondary_group evaluation: Resolved