[YARN-9698] [Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.4.0
• Component/s: capacity scheduler
• Labels: fs2cs
• Assignee: Weiwei Yang
• Reporter: Weiwei Yang
• Created: 25/Jul/19 07:24
• Updated: 11/Feb/24 09:10
• Resolved: 06/Jul/21 11:44

Description

We see some users want to migrate from Fair Scheduler to Capacity Scheduler, this Jira is created as an umbrella to track all related efforts for the migration, the scope contains

• Bug fixes
• Add missing features
• Migration tools that help to generate CS configs based on FS, validate configs etc
• Documents

this is part of CS component, the purpose is to make the migration process smooth.

Issue Links

• incorporates: YARN-10623 Capacity scheduler should support refresh queue automatically by a thread policy.
• is a parent of: YARN-9748 Allow capacity-scheduler configuration on HDFS and support reload from HDFS
• is blocked by: YARN-10546 Limit application resource reservation on nodes for non-node/rack specific requests shoud be supported in CS.
• relates to: YARN-9866 u:user2:%primary_group is not working as expected
• relates to: YARN-9867 "u:%user:parentqueue.%user" is not working as expected
• relates to: YARN-9868 Validate %primary_group queue in CS queue manager
• relates to: YARN-10610 Add queuePath to RESTful API for CapacityScheduler consistent with FairScheduler queuePath

Sub-Tasks

• Migration tool that help to generate CS config based on FS config [Phase 1]: Resolved
• Migration tool that help to generate CS config based on FS config [Phase 2]: Resolved
• RM fails to start with FileSystemBasedConfigurationProvider: Resolved
• Allow multiple leaf queues with the same name in CapacityScheduler: Resolved
• Capacity scheduler: add support for Secondary Group rule mapping: Resolved
• Capacity scheduler: add support for combined %user + %primary_group mapping: Resolved
• Capacity scheduler: add support for combined %user + %secondary_group mapping: Resolved
• Capacity scheduler: add support for limiting maxRunningApps per user: Resolved
• Capacity scheduler: add support for default maxRunningApps limit per user: Resolved
• Capacity scheduler: enhance capacity / maximum-capacity setting: Resolved
• Capacity scheduler: enhance leaf-queue-template capacity / maximum-capacity setting: Resolved
• Fix JavaDoc errors introduced by YARN-9699: Resolved
• Support max running app logic for CapacityScheduler: Resolved
• FairOrderingPolicy Improvements: Resolved
• Add dry-run feature to FS-CS converter tool: Resolved
• FS-CS converter: disable terminal placement rule checking: Resolved
• FS-CS converter: remove mixed ordering policy check: Resolved
• FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues: Resolved
• Capacity scheduler: add support for %specified mapping: Resolved
• Capacity scheduler: add support for create=true/false per mapping rule: Resolved
• FS-CS converter: dry run should work without output defined: Resolved
• FS-CS converter: separator between mapping rules should be comma: Resolved
• FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping: Resolved
• FS-CS converter: adjust queue ACL to have the same output with CS as for FS has: Resolved
• FSQueueConverter should not set App Ordering Policy to Parent Queue: Resolved
• FS-CS converter: Do not allow output dir to be the same as input dir: Resolved
• FS-CS converter tool: issue warning on dynamic auto-create mapping rules: Resolved
• FS-CS converter: initPropertyActions() is not called without rules file: Resolved
• FS-CS converter: convert property yarn.scheduler.fair.update-interval-ms: Resolved
• FS-CS Converter: Need to validate c-s.xml after converting: Resolved
• FS-CS Converter: tool doesn't handle min/max resource conversion correctly: Resolved
• FS-CS converter: only convert placement rules if a cmd line switch is defined: Resolved
• FS-CS converter: call System.exit function call for every code path in main method: Resolved
• FS-CS converter: fix incorrect capacity conversion: Resolved
• FS-CS converter: fix emitted ordering policy string and max-am-resource percent value: Resolved
• FS-CS converter: don't enable auto-create queue property for root: Resolved
• FS-CS converter: skip increment properties for mem/vcores and fix DRF check: Resolved
• FS-CS converter: add an option to enable asynchronous scheduling in CapacityScheduler: Resolved
• FS-CS converter: convert maxAppsDefault, maxRunningApps settings: Resolved
• Document max-parallel-apps for Capacity Scheduler: Resolved
• FS-CS converter: handle case when weight=0 and allow more lenient capacity checks in Capacity Scheduler: Resolved
• Remove "experimental" warning message from fs2cs: Resolved
• fs2cs: add queue autorefresh policy during conversion: Resolved